Systems and methods for vehicle braking

ABSTRACT

Systems and methods use cameras to provide autonomous navigation features. In one implementation, a method for navigating a user vehicle may include acquiring, using at least one image capture device, a plurality of images of an area in a vicinity of the user vehicle; determining from the plurality of images a first lane constraint on a first side of the user vehicle and a second lane constraint on a second side of the user vehicle opposite to the first side of the user vehicle; enabling the user vehicle to pass a target vehicle if the target vehicle is determined to be in a lane different from the lane in which the user vehicle is traveling; and causing the user vehicle to abort the pass before completion of the pass, if the target vehicle is determined to be entering the lane in which the user vehicle is traveling.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 16/691,177, filed Nov. 21, 2019, which is a continuation applicationof U.S. application Ser. No. 16/393,846, filed Apr. 24, 2019, which is acontinuation of U.S. application Ser. No. 14/560,420, filed Dec. 4,2014, now U.S. Pat. No. 10,293,826, issued May 21, 2019, which claimsthe benefit of priority of U.S. Provisional Patent Application No.61/911,490, filed Dec. 4, 2013, U.S. Provisional Patent Application No.61/993,084, filed on May 14, 2014, U.S. Provisional Patent ApplicationNo. 61/993,111, filed on May 14, 2014; U.S. Provisional PatentApplication 62/015,524, filed on Jun. 23, 2014; U.S. Provisional PatentApplication 62/022,221, filed on Jul. 9, 2014; U.S. Provisional PatentApplication No. 62/040,224, filed on Aug. 21, 2014; and U.S. ProvisionalPatent Application 62/040,269, filed on Aug. 21, 2014. All of theforegoing applications are incorporated herein by reference in theirentirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to autonomous vehiclenavigation and, more specifically, to systems and methods that usecameras to provide autonomous vehicle navigation features.

II. Background Information

As technology continues to advance, the goal of a fully autonomousvehicle that is capable of navigating on roadways is on the horizon.Primarily, an autonomous vehicle may be able to identify its environmentand navigate without input from a human operator. Autonomous vehiclesmay also take into account a variety of factors and make appropriatedecisions based on those factors to safely and accurately reach anintended destination. For example, various objects—such as othervehicles and pedestrians—are encountered when a vehicle typicallytravels a roadway. Autonomous driving systems may recognize theseobjects in a vehicle's environment and take appropriate and timelyaction to avoid collisions. Additionally, autonomous driving systems mayidentify other indicators—such as traffic signals, traffic signs, andlane markings—that regulate vehicle movement (e.g., when the vehiclemust stop and may go, a speed at which the vehicle must not exceed,where the vehicle must be positioned on the roadway, etc.). Autonomousdriving systems may need to determine when a vehicle should changelanes, turn at intersections, change roadways, etc. As is evident fromthese examples, many factors may need to be addressed in order toprovide an autonomous vehicle that is capable of navigating safely andaccurately.

SUMMARY

Embodiments consistent with the present disclosure provide systems andmethods for autonomous vehicle navigation. The disclosed embodiments mayuse cameras to provide autonomous vehicle navigation features. Forexample, consistent with the disclosed embodiments, the disclosedsystems may include one, two, or more cameras that monitor theenvironment of a vehicle and cause a navigational response based on ananalysis of images captured by one or more of the cameras.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a vehicle. The system may include at least oneimage capture device configured to acquire a plurality of images of anarea in a vicinity of the vehicle; a data interface; and at least oneprocessing device. The at least one processing device may be configuredto: receive the plurality of images via the data interface; determinefrom the plurality of images a first lane constraint on a first side ofthe vehicle; determine from the plurality of images a second laneconstraint on a second side of the vehicle opposite to the first side ofthe vehicle, wherein the first and second lane constraints define a lanewithin which the vehicle travels and wherein a first distancecorresponds to a distance between the first side of the vehicle and thefirst lane constraint and a second distance corresponds to a distancebetween the second side of the vehicle and the second lane constraint;determine, based on the plurality of images, whether a lane offsetcondition exists on the first side of the vehicle; if a lane offsetcondition exists on the first side of the vehicle, cause the vehicle totravel within the first and second lane constraints such that the firstdistance is greater than the second distance; determine, based on theplurality of images, whether a lane offset condition exists on thesecond side of the vehicle; and if a lane offset condition exists on thesecond side of the vehicle, cause the vehicle to travel within the firstand second lane constraints such that the first distance is less thanthe second distance.

Consistent with another disclosed embodiment, a vehicle may include afirst vehicle side; a second vehicle side opposite the first vehicleside; at least one image capture device configured to acquire aplurality of images of an area in a vicinity of the vehicle; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; determine from the plurality of images a firstlane constraint on the first vehicle side; determine from the pluralityof images a second lane constraint on the second vehicle side, whereinthe first and second lane constraints define a lane within which thevehicle travels and wherein a first distance corresponds to a distancebetween the first vehicle side and the first lane constraint and asecond distance corresponds to a distance between the second vehicleside and the second lane constraint; determine, based on the pluralityof images, whether a lane offset condition exists on the first vehicleside; if a lane offset condition exists on the first vehicle side, causethe vehicle to travel within the first and second lane constraints suchthat the first distance is greater than the second distance; determine,based on the plurality of images, whether a lane offset condition existson the second vehicle side; and if a lane offset condition exists on thesecond vehicle side, cause the vehicle to travel within the first andsecond lane constraints such that the first distance is less than thesecond distance.

Consistent with another disclosed embodiment, a method is provided fornavigating a vehicle. The method may include acquiring, using at leastone image capture device, a plurality of images of an area in thevicinity of the vehicle; determining from the plurality of images afirst lane constraint on a first side of the vehicle; determining fromthe plurality of images a second lane constraint on a second side of thevehicle opposite to the first side of the vehicle, wherein the first andsecond lane constraints define a lane within which the vehicle travelsand wherein a first distance corresponds to a distance between the firstside of the vehicle and the first lane constraint and a second distancecorresponds to a distance between the second side of the vehicle and thesecond lane constraint; determining, based on the plurality of images,whether a lane offset condition exists on the first side of the vehicle;if a lane offset condition exists on the first side of the vehicle,causing the vehicle to travel within the first and second laneconstraints such that the first distance is greater than the seconddistance; determining, based on the plurality of images, whether a laneoffset condition exists on the second side of the vehicle; and if a laneoffset condition exists on the second side of the vehicle, causing thevehicle to travel within the first and second lane constraints such thatthe first distance is less than the second distance.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a vehicle. The system may include at least oneimage capture device configured to acquire a plurality of images of anarea in a vicinity of the vehicle; a data interface; and at least oneprocessing device. The at least processing device may be configured to:receive the plurality of images via the data interface; determine fromthe plurality of images a current lane of travel from among a pluralityof available travel lanes; and cause the vehicle to change lanes if thecurrent lane of travel is not the same as a predetermined default travellane.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire aplurality of images of an area in a vicinity of the vehicle; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; determine from the plurality of images a currentlane of travel from among a plurality of available travel lanes; andcause the vehicle to change lanes if the current lane of travel is notthe same as a predetermined default travel lane.

Consistent with another disclosed embodiment, a method is provided fornavigating a vehicle. The method may include acquiring, using at leastone image capture device, a plurality of images of an area in a vicinityof the vehicle; determining from the plurality of images a current laneof travel from among a plurality of available travel lanes; and causingthe vehicle to change lanes if the current lane of travel is not thesame as a predetermined default travel lane.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a vehicle. The system may include at least oneimage capture device configured to acquire a plurality of images of anarea in a vicinity of the vehicle; a data interface; and at least oneprocessing device. The at least one processing device may be configuredto: receive the plurality of images via the data interface; recognize acurve to be navigated based on map data and vehicle positioninformation; determine an initial target velocity for the vehicle basedon at least one characteristic of the curve as reflected in the mapdata; adjust a velocity of the vehicle to the initial target velocity;determine, based on the plurality of images, one or more observedcharacteristics of the curve; determine an updated target velocity basedon the one or more observed characteristics of the curve; and adjust thevelocity of the vehicle to the updated target velocity.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire aplurality of images of an area in a vicinity of the vehicle; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; recognize a curve to be navigated based on mapdata and vehicle position information; determine an initial targetvelocity for the vehicle based on at least one characteristic of thecurve as reflected in the map data; adjust a velocity of the vehicle tothe initial target velocity; determine, based on the plurality ofimages, one or more observed characteristics of the curve; determine anupdated target velocity based on the one or more observedcharacteristics of the curve; and adjust the velocity of the vehicle tothe updated target velocity.

Consistent with another disclosed embodiment, a method is provided fornavigating a vehicle. The method may include acquiring, using at leastone image capture device, a plurality of images of an area in a vicinityof the vehicle; recognizing a curve to be navigated based on map dataand vehicle position information; determining an initial target velocityfor the vehicle based on at least one characteristic of the curve asreflected in the map data; adjusting a velocity of the vehicle to theinitial target velocity; determining, based on the plurality of images,one or more observed characteristics of the curve; determining anupdated target velocity based on the one or more observedcharacteristics of the curve; and adjusting the velocity of the vehicleto the updated target velocity.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a primary vehicle. The system may include atleast one image capture device configured to acquire a plurality ofimages of an area in a vicinity of the primary vehicle; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; determine, from at least some of the pluralityof images, a first lane constraint on a first side of the primaryvehicle; determine, from at least some of the plurality of images, asecond lane constraint on a second side of the primary vehicle oppositeto the first side of the primary vehicle, wherein the first and secondlane constraints define a lane within which the primary vehicle travels;cause the primary vehicle to travel within the first and second laneconstraints; locate in the plurality of images a leading vehicle;determine, based on the plurality of images, at least one action takenby the leading vehicle; and cause the primary vehicle to mimic the atleast one action of the leading vehicle.

Consistent with another disclosed embodiment, a primary vehicle mayinclude a body; at least one image capture device configured to acquirea plurality of images of an area in a vicinity of the primary vehicle; adata interface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; determine, from at least some of the pluralityof images, a first lane constraint on a first side of the primaryvehicle; determine, from at least some of the plurality of images, asecond lane constraint on a second side of the primary vehicle oppositeto the first side of the primary vehicle, wherein the first and secondlane constraints define a lane within which the primary vehicle travels;cause the primary vehicle to travel within the first and second laneconstraints; locate in the plurality of images a leading vehicle;determine, based on the plurality of images, at least one action takenby the leading vehicle; and cause the primary vehicle to mimic the atleast one action of the leading vehicle.

Consistent with another disclosed embodiment, a method is provided fornavigating a primary vehicle. The method ma include acquiring, using atleast one image capture device, a plurality of images of an area in avicinity of the primary vehicle; determining, from at least some of theplurality of images, a first lane constraint on a first side of theprimary vehicle; determining, from at least some of the plurality ofimages, a second lane constraint on a second side of the primary vehicleopposite to the first side of the primary vehicle, wherein the first andsecond lane constraints define a lane within which the primary vehicletravels; causing the primary vehicle to travel within the first andsecond lane constraints; locating in the plurality of images a leadingvehicle; determining, based on the plurality of images, at least oneaction taken by the leading vehicle; and causing the primary vehicle tomimic the at least one action of the leading vehicle.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a user vehicle. They system may include at leastone image capture device configured to acquire a plurality of images ofan area in a vicinity of the user vehicle; a data interface; and atleast one processing device. The at least one processing device may beconfigured to: receive the plurality of images via the data interface;determine from the plurality of images a first lane constraint on afirst side of the user vehicle; determine from the plurality of images asecond lane constraint on a second side of the user vehicle opposite tothe first side of the user vehicle, wherein the first and second laneconstraints define a lane within which the user vehicle travels;acquire, based on the plurality of images, a target vehicle; enable theuser vehicle to pass the target vehicle if the target vehicle isdetermined to be in a lane different from the lane in which the uservehicle is traveling; monitor a position of the target vehicle based onthe plurality of images; and cause the user vehicle to abort the passbefore completion of the pass, if the target vehicle is determined to beentering the lane in which the user vehicle is traveling.

Consistent with another disclosed embodiment, a user vehicle may includeat least one image capture device configured to acquire a plurality ofimages of an area in a vicinity of the user vehicle; a data interface;and at least one processing device. The at least one processing devicemay be configured to: receive the plurality of images via the datainterface; determine from the plurality of images a first laneconstraint on a first side of the user vehicle; determine from theplurality of images a second lane constraint on a second side of theuser vehicle opposite to the first side of the user vehicle, wherein thefirst and second lane constraints define a lane within which the uservehicle travels; acquire, based on the plurality of images, a targetvehicle; enable the user vehicle to pass the target vehicle if thetarget vehicle is determined to be in a lane different from the lane inwhich the user vehicle is traveling; monitor a position of the targetvehicle based on the plurality of images; and cause the user vehicle toabort the pass before completion of the pass, if the target vehicle isdetermined to be entering the lane in which the user vehicle istraveling.

Consistent with another disclosed embodiment, a method is provided fornavigating a user vehicle. The method may include acquiring, using atleast one image capture device, a plurality of images of an area in avicinity of the user vehicle; determining from the plurality of images afirst lane constraint on a first side of the user vehicle; determiningfrom the plurality of images a second lane constraint on a second sideof the user vehicle opposite to the first side of the user vehicle,wherein the first and second lane constraints define a lane within whichthe user vehicle travels; acquiring, based on the plurality of images, atarget vehicle; enabling the user vehicle to pass the target vehicle ifthe target vehicle is determined to be in a lane different from the lanein which the user vehicle is traveling; monitoring a position of thetarget vehicle based on the plurality of images; and causing the uservehicle to abort the pass before completion of the pass, if the targetvehicle is determined to be entering the lane in which the user vehicleis traveling.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a user vehicle. The system may include at leastone image capture device configured to acquire a plurality of images ofan area in a vicinity of the user vehicle; a data interface; and atleast one processing device. The at least one processing device may beconfigured to: receive the plurality of images via the data interface;determine from the plurality of images a first lane constraint on afirst side of the user vehicle; determine from the plurality of images asecond lane constraint on a second side of the user vehicle opposite tothe first side of the user vehicle, wherein the first and second laneconstraints define a lane within which the user vehicle travels;determine, based on the plurality of images, whether an encroachingvehicle is approaching from a side of the user vehicle; and cause theuser vehicle to maintain a current velocity and to travel within thefirst and second lane constraints such that a first distance, on theside of the user vehicle that the encroaching vehicle is approachingfrom, is greater than a second distance.

Consistent with another disclosed embodiment, a user vehicle may includeat least one image capture device configured to acquire a plurality ofimages of an area in a vicinity of the user vehicle; a data interface;and at least one processing device. The at least one processing devicemay be configured to: receive the plurality of images via the datainterface; determine from the plurality of images a first laneconstraint on a first side of the user vehicle; determine from theplurality of images a second lane constraint on a second side of theuser vehicle opposite to the first side of the user vehicle, wherein thefirst and second lane constraints define a lane within which the uservehicle travels; determine, based on the plurality of images, whether anencroaching vehicle is approaching from a side of the user vehicle; andcause the user vehicle to maintain a current velocity and to travelwithin the first and second lane constraints such that a first distance,on the side of the user vehicle that the encroaching vehicle isapproaching from, is greater than a second distance.

Consistent with another disclosed embodiment, a method for navigating auser vehicle is provided. The method may include acquiring, using atleast one image capture device, a plurality of images of an area in avicinity of the user vehicle; determining from the plurality of images afirst lane constraint on a first side of the user vehicle; determiningfrom the plurality of images a second lane constraint on a second sideof the user vehicle opposite to the first side of the user vehicle,wherein the first and second lane constraints define a lane within whichthe user vehicle travels; determining, based on the plurality of images,whether an encroaching vehicle is approaching from a side of the uservehicle; and causing the user vehicle to maintain a current velocity andto travel within the first and second lane constraints such that a firstdistance, on the side of the user vehicle that the encroaching vehicleis approaching from, is greater than a second distance.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a primary vehicle. They system may include atleast one image capture device configured to acquire a plurality ofimages of an area in a vicinity of the primary vehicle; a datainterface; and at least one processing device. The at least processingdevice may be configured to: receive the plurality of images via thedata interface; identify a target object within the plurality of images;monitor, via the plurality of images, a motion of the target object anda distance between the primary vehicle and the target object; determinean indicator of an intercept time between the primary vehicle and thetarget object based on the monitored motion and the distance between theprimary vehicle and the target object; and cause a response in theprimary vehicle based on a comparison of the intercept time to aplurality of predetermined intercept thresholds.

Consistent with another disclosed embodiment, a primary vehicle mayinclude a body; at least one image capture device configured to acquirea plurality of images of an area in a vicinity of the primary vehicle; adata interface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; identify a target object within the plurality ofimages; monitor, via the plurality of images, a motion of the targetobject and a distance between the primary vehicle and the target object;determine an indicator of an intercept time between the primary vehicleand the target object based on the monitored motion and the distancebetween the primary vehicle and the target object; and cause a responsein the primary vehicle based on a comparison of the intercept time to aplurality of predetermined intercept thresholds.

Consistent with another disclosed embodiment, a method for navigating aprimary vehicle may include acquiring, via at least one image capturedevice, a plurality of images of an area in a vicinity of the primaryvehicle; identifying a target object within the plurality of images;monitoring, based on the plurality of images, a motion of the targetobject and a distance between the primary vehicle and the target object;determining an indicator of an intercept time between the primaryvehicle and the target object based on the monitored motion and thedistance between the primary vehicle and the target object; and causinga response in the primary vehicle based on a comparison of the intercepttime to a plurality of predetermined intercept thresholds.

Consistent with a disclosed embodiment, a driver assist navigationsystem is provided for a vehicle. The system may include at least oneimage capture device configured to acquire a plurality of images of anarea in a vicinity of the vehicle; a data interface; and at least oneprocessing device. The at least one processing device may be configuredto: receive the plurality of images via the data interface; identify,based on analysis of the plurality of images, a trigger for stopping thevehicle; and based on the identified trigger, cause the vehicle to stopaccording to a braking profile including a first segment associated witha first deceleration rate, a second segment which includes a seconddeceleration rate less than the first deceleration rate, and a thirdsegment in which a level of braking is decreased as a target stoppinglocation is approached, as determined based on the analysis of theplurality of images.

Consistent with another disclosed embodiment, a vehicle may include abody; at least one image capture device configured to acquire aplurality of images of an area in a vicinity of the vehicle; a datainterface; and at least one processing device. The at least oneprocessing device may be configured to: receive the plurality of imagesvia the data interface; identify, based on analysis of the plurality ofimages, a trigger for stopping the vehicle; and based on the identifiedtrigger, cause the vehicle to stop according to a braking profileincluding a first segment associated with a first deceleration rate, asecond segment which includes a second deceleration rate less than thefirst deceleration rate, and a third segment in which a level of brakingis decreased as a target stopping location is approached, as determinedbased on the analysis of the plurality of images.

Consistent with another disclosed embodiment, a method is provided fornavigating a vehicle. The method may include acquiring, via at least oneimage capture plurality of images of an area in a vicinity of thevehicle; identifying, based on analysis of the plurality of images, atrigger for stopping the vehicle; and based on the identified trigger,causing the vehicle to stop according to a braking profile including afirst segment associated with a first deceleration rate, a secondsegment which includes a second deceleration rate less than the firstdeceleration rate, and a third segment in which a level of braking isdecreased as a target stopping location is approached, as determinedbased on the analysis of the plurality of images.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a diagrammatic representation of an exemplary systemconsistent with the disclosed embodiments.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments.

FIG. 2B is a diagrammatic top view representation of the vehicle andsystem shown in FIG. 2A consistent with the disclosed embodiments.

FIG. 2C is a diagrammatic top view representation of another embodimentof a vehicle including a system consistent with the disclosedembodiments.

FIG. 2D is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2E is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems consistent with the disclosed embodiments.

FIG. 3A is a diagrammatic representation of an interior of a vehicleincluding a rearview mirror and a user interface for a vehicle imagingsystem consistent with the disclosed embodiments.

FIG. 3B is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 3C is an illustration of the camera mount shown in FIG. 3B from adifferent perspective consistent with the disclosed embodiments.

FIG. 3D is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 4 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 5A is a flowchart showing an exemplary process for causing one ormore navigational responses based on monocular image analysis consistentwith disclosed embodiments.

FIG. 5B is a flowchart showing an exemplary process for detecting one ormore vehicles and/or pedestrians in a set of images consistent with thedisclosed embodiments.

FIG. 5C is a flowchart showing an exemplary process for detecting roadmarks and/or lane geometry information in a set of images consistentwith the disclosed embodiments.

FIG. 5D is a flowchart showing an exemplary process for detectingtraffic lights in a set of images consistent with the disclosedembodiments.

FIG. 5E is a flowchart showing an exemplary process for causing one ormore navigational responses based on a vehicle path consistent with thedisclosed embodiments.

FIG. 5F is a flowchart showing an exemplary process for determiningwhether a leading vehicle is changing lanes consistent with thedisclosed embodiments.

FIG. 6 is a flowchart showing an exemplary process for causing one ormore navigational responses based on stereo image analysis consistentwith the disclosed embodiments.

FIG. 7 is a flowchart showing an exemplary process for causing one ormore navigational responses based on an analysis of three sets of imagesconsistent with the disclosed embodiments.

FIG. 8A is a diagrammatic representation of an exemplary vehicletraveling within a first and a second lane constraint in which a laneoffset condition exists on the first vehicle side consistent with thedisclosed embodiments.

FIG. 8B is a diagrammatic representation of an exemplary vehicletraveling within a first and a second lane constraint in which a laneoffset condition exists on the second vehicle side consistent with thedisclosed embodiments.

FIG. 8C is a diagrammatic representation of an exemplary vehicletraveling within a first and a second lane constraint on a curved roadconsistent with the disclosed embodiments.

FIG. 8D is a diagrammatic representation of an exemplary vehicletraveling within a first and a second lane constraint in which a laneoffset condition exists on both the first vehicle side and on the secondvehicle side consistent with the disclosed embodiments.

FIG. 9 is a diagrammatic representation of the memory of an exemplarynavigation system consistent with the disclosed embodiments.

FIG. 10 is a flowchart of an exemplary method for navigating a vehicleconsistent with the disclosed embodiments.

FIG. 11 is a diagrammatic representation of an exemplary vehicleincluding a navigation system traveling within a default lane consistentwith the disclosed embodiments.

FIG. 12A is a diagrammatic representation of the memory of an exemplarynavigation system consistent with the disclosed embodiments.

FIG. 12B is a flowchart of an exemplary process for navigating a vehicleconsistent with the disclosed embodiments.

FIGS. 13A and 13B are diagrammatic representations of an exemplaryvehicle approaching, and navigating, a curve with one or morecharacteristics consistent with the disclosed embodiments.

FIG. 14 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 15 is a flow chart showing an exemplary process for controlling thevelocity of a vehicle based on a detected curve and observedcharacteristics of the curve consistent with disclosed embodiments.

FIGS. 16A-16D are diagrammatic representations of a primary vehiclemimicking one or more actions of a leading vehicle consistent with thedisclosed embodiments.

FIG. 17 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 18 is a flow chart showing an exemplary process for causing aprimary vehicle to mimic one or more actions of a leading vehicleconsistent with disclosed embodiments.

FIG. 19 is a flow chart showing an exemplary process for causing aprimary vehicle to decline to mimic a turn of a leading vehicleconsistent with disclosed embodiments.

FIG. 20 is a flow chart showing an exemplary process for causing aprimary vehicle to mimic or decline a turn of a leading vehicleconsistent with disclosed embodiments.

FIG. 21 is a flow chart showing another exemplary process for causing aprimary vehicle to mimic one or more actions of a leading vehicleconsistent with disclosed embodiments.

FIG. 22 is a flow chart showing an exemplary process for causing aprimary vehicle to mimic one or more actions of a leading vehicle basedon a navigation history consistent with disclosed embodiments.

FIG. 23 is a diagrammatic representation of an exemplary vehicletraveling within a first and a second lane constraint on a roadway alongon which another vehicle is traveling consistent with the disclosedembodiments.

FIG. 24A is a diagrammatic representation of a memory storinginstructions for navigating a vehicle among encroaching vehiclesconsistent with the disclosed embodiments.

FIG. 24B is a flowchart of an exemplary process for navigating a vehicleamong encroaching vehicles consistent with the disclosed embodiments.

FIG. 25 is a diagrammatic representation of an exemplary vehicletraveling within a first and a second lane constraint on a roadway alongon which another vehicle is traveling consistent with the disclosedembodiments.

FIG. 26 is a diagrammatic representation of a memory storinginstructions for detecting and responding to traffic laterallyencroaching on a vehicle consistent with the disclosed embodiments.

FIG. 27 is a flowchart of an exemplary process for detecting andresponding to traffic laterally encroaching on a vehicle consistent withthe disclosed embodiments.

FIGS. 28A and 28B are diagrammatic representations of causing a responsein a primary vehicle consistent with the disclosed embodiments.

FIG. 29 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 30 is a flow chart showing an exemplary process for causing aresponse in a primary vehicle consistent with disclosed embodiments.

FIG. 31 is a flow chart showing an exemplary process for declining tocause a response in a primary vehicle consistent with disclosedembodiments.

FIG. 32A is a diagrammatic representation of an exemplary vehicle inwhich an object is present in the vicinity of vehicle consistent withthe disclosed embodiments.

FIG. 32B illustrates an exemplary braking profile, consistent with thedisclosed embodiments.

FIG. 33 is a diagrammatic representation of the memory of an exemplarynavigation system consistent with the disclosed embodiments.

FIG. 34 is a flowchart of an exemplary method for implementing a brakingprofile while navigating a vehicle consistent with the disclosedembodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

Disclosed embodiments provide systems and methods that use cameras toprovide autonomous navigation features. In various embodiments, thesystem may include one, two or more cameras that monitor the environmentof a vehicle. In one embodiment, the system may monitor and adjust thefree space between a vehicle and the boundaries of the lane within whichthe vehicle is traveling. In another embodiment, the system may select aparticular lane as a default lane for the vehicle to use whiletraveling. In another embodiment, the system may control the speed ofthe vehicle in different scenarios, such as while making a turn. In yetanother embodiment, the system may mimic the actions of a leadingvehicle. In yet another embodiment, the system may monitor a targetvehicle and enable the vehicle to pass the target vehicle under certainconditions (e.g., if the target vehicle is traveling in a lane differentfrom the lane within which the vehicle is traveling. In still yetanother embodiment, the system may produce a natural feeling response toa laterally encroaching vehicle, such as a vehicle attempting to moveinto the lane within which the vehicle is traveling.

FIG. 1 is a block diagram representation of a system 100 consistent withthe exemplary disclosed embodiments. System 100 may include variouscomponents depending on the requirements of a particular implementation.In some embodiments, system 100 may include a processing unit 110, animage acquisition unit 120, a position sensor 130, one or more memoryunits 140, 150, a map database 160, and a user interface 170. Processingunit 110 may include one or more processing devices. In someembodiments, processing unit 110 may include an applications processor180, an image processor 190, or any other suitable processing device.Similarly, image acquisition unit 120 may include any number of imageacquisition devices and components depending on the requirements of aparticular application. In some embodiments, image acquisition unit 120may include one or more image capture devices (e.g., cameras), such asimage capture device 122, image capture device 124, and image capturedevice 126. System 100 may also include a data interface 128communicatively connecting processing device 110 to image acquisitiondevice 120. For example, data interface 128 may include any wired and/orwireless link or links for transmitting image data acquired by imageaccusation device 120 to processing unit 110.

Both applications processor 180 and image processor 190 may includevarious types of processing devices. For example, either or both ofapplications processor 180 and image processor 190 may include amicroprocessor, preprocessors (such as an image preprocessor), graphicsprocessors, a central processing unit (CPU), support circuits, digitalsignal processors, integrated circuits, memory, or any other types ofdevices suitable for running applications and for image processing andanalysis. In some embodiments, applications processor 180 and/or imageprocessor 190 may include any type of single or multi-core processor,mobile device microcontroller, central processing unit, etc. Variousprocessing devices may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc. and may includevarious architectures (e.g., x86 processor, ARM®, etc.).

In some embodiments, applications processor 180 and/or image processor190 may include any of the EyeQ series of processor chips available fromMobileye®. These processor designs each include multiple processingunits with local memory and instruction sets. Such processors mayinclude video inputs for receiving image data from multiple imagesensors and may also include video out capabilities. In one example, theEyeQ2® uses 90 nm-micron technology operating at 332 Mhz. The EyeQ2®architecture consists of two floating point, hyper-thread 32-bit RISCCPUs (MIPS32® 34K® cores), five Vision Computing Engines (VCE), threeVector Microcode Processors (VMP®), Denali 64-bit Mobile DDR Controller,128-bit internal Sonics Interconnect, dual 16-bit Video input and 18-bitVideo output controllers, 16 channels DMA and several peripherals. TheMIPS34K CPU manages the five VCEs, three VMP™ and the DMA, the secondMIPS34K CPU and the multi-channel DMA as well as the other peripherals.The five VCEs, three VMP® and the MIPS34K CPU can perform intensivevision computations required by multi-function bundle applications. Inanother example, the EyeQ3®, which is a third generation processor andis six times more powerful that the EyeQ2®, may be used in the disclosedembodiments.

While FIG. 1 depicts two separate processing devices included inprocessing unit 110, more or fewer processing devices may be used. Forexample, in some embodiments, a single processing device may be used toaccomplish the tasks of applications processor 180 and image processor190. In other embodiments, these tasks may be performed by more than twoprocessing devices.

Processing unit 110 may comprise various types of devices. For example,processing unit 110 may include various devices, such as a controller,an image preprocessor, a central processing unit (CPU), supportcircuits, digital signal processors, integrated circuits, memory, or anyother types of devices for image processing and analysis. The imagepreprocessor may include a video processor for capturing, digitizing andprocessing the imagery from the image sensors. The CPU may comprise anynumber of microcontrollers or microprocessors. The support circuits maybe any number of circuits generally well known in the art, includingcache, power supply, clock and input-output circuits. The memory maystore software that, when executed by the processor, controls theoperation of the system. The memory may include databases and imageprocessing software. The memory may comprise any number of random accessmemories, read only memories, flash memories, disk drives, opticalstorage, tape storage, removable storage and other types of storage. Inone instance, the memory may be separate from the processing unit 110.In another instance, the memory may be integrated into the processingunit 110.

Each memory 140, 150 may include software instructions that whenexecuted by a processor (e.g., applications processor 180 and/or imageprocessor 190), may control operation of various aspects of system 100.These memory units may include various databases and image processingsoftware. The memory units may include random access memory, read onlymemory, flash memory, disk drives, optical storage, tape storage,removable storage and/or any other types of storage. In someembodiments, memory units 140, 150 may be separate from the applicationsprocessor 180 and/or image processor 190. In other embodiments, thesememory units may be integrated into applications processor 180 and/orimage processor 190.

Position sensor 130 may include any type of device suitable fordetermining a location associated with at least one component of system100. In some embodiments, position sensor 130 may include a GPSreceiver. Such receivers can determine a user position and velocity byprocessing signals broadcasted by global positioning system satellites.Position information from position sensor 130 may be made available toapplications processor 180 and/or image processor 190.

User interface 170 may include any device suitable for providinginformation to or for receiving inputs from one or more users of system100. In some embodiments, user interface 170 may include user inputdevices, including, for example, a touchscreen, microphone, keyboard,pointer devices, track wheels, cameras, knobs, buttons, etc. With suchinput devices, a user may be able to provide information inputs orcommands to system 100 by typing instructions or information, providingvoice commands, selecting menu options on a screen using buttons,pointers, or eye-tracking capabilities, or through any other suitabletechniques for communicating information to system 100.

User interface 170 may be equipped with one or more processing devicesconfigured to provide and receive information to or from a user andprocess that information for use by, for example, applications processor180. In some embodiments, such processing devices may executeinstructions for recognizing and tracking eye movements, receiving andinterpreting voice commands, recognizing and interpreting touches and/orgestures made on a touchscreen, responding to keyboard entries or menuselections, etc. In some embodiments, user interface 170 may include adisplay, speaker, tactile device, and/or any other devices for providingoutput information to a user.

Map database 160 may include any type of database for storing map datauseful to system 100. In some embodiments, map database 160 may includedata relating to the position, in a reference coordinate system, ofvarious items, including roads, water features, geographic features,businesses, points of interest, restaurants, gas stations, etc. Mapdatabase 160 may store not only the locations of such items, but alsodescriptors relating to those items, including, for example, namesassociated with any of the stored features. In some embodiments, mapdatabase 160 may be physically located with other components of system100. Alternatively or additionally, map database 160 or a portionthereof may be located remotely with respect to other components ofsystem 100 (e.g., processing unit 110). In such embodiments, informationfrom map database 160 may be downloaded over a wired or wireless dataconnection to a network (e.g., over a cellular network and/or theInternet, etc.).

Image capture devices 122, 124, and 126 may each include any type ofdevice suitable for capturing at least one image from an environment.Moreover, any number of image capture devices may be used to acquireimages for input to the image processor. Some embodiments may includeonly a single image capture device, while other embodiments may includetwo, three, or even four or more image capture devices. Image capturedevices 122, 124, and 126 will be further described with reference toFIGS. 2B-2E, below.

System 100, or various components thereof, may be incorporated intovarious different platforms. In some embodiments, system 100 may beincluded on a vehicle 200, as shown in FIG. 2A. For example, vehicle 200may be equipped with a processing unit 110 and any of the othercomponents of system 100, as described above relative to FIG. 1 . Whilein some embodiments vehicle 200 may be equipped with only a single imagecapture device (e.g., camera), in other embodiments, such as thosediscussed in connection with FIGS. 2B-2E, multiple image capture devicesmay be used. For example, either of image capture devices 122 and 124 ofvehicle 200, as shown in FIG. 2A, may be part of an ADAS (AdvancedDriver Assistance Systems) imaging set.

The image capture devices included on vehicle 200 as part of the imageacquisition unit 120 may be positioned at any suitable location. In someembodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122may be located in the vicinity of the rearview mirror. This position mayprovide a line of sight similar to that of the driver of vehicle 200,which may aid in determining what is and is not visible to the driver.Image capture device 122 may be positioned at any location near therearview mirror, but placing image capture device 122 on the driver sideof the mirror may further aid in obtaining images representative of thedriver's field of view and/or line of sight.

Other locations for the image capture devices of image acquisition unit120 may also be used. For example, image capture device 124 may belocated on or in a bumper of vehicle 200. Such a location may beespecially suitable for image capture devices having a wide field ofview. The line of sight of bumper-located image capture devices can bedifferent from that of the driver and, therefore, the bumper imagecapture device and driver may not always see the same objects. The imagecapture devices (e.g., image capture devices 122, 124, and 126) may alsobe located in other locations. For example, the image capture devicesmay be located on or in one or both of the side mirrors of vehicle 200,on the roof of vehicle 200, on the hood of vehicle 200, on the trunk ofvehicle 200, on the sides of vehicle 200, mounted on, positioned behind,or positioned in front of any of the windows of vehicle 200, and mountedin or near light figures on the front and/or back of vehicle 200, etc.

In addition to image capture devices, vehicle 200 may include variousother components of system 100. For example, processing unit 110 may beincluded on vehicle 200 either integrated with or separate from anengine control unit (ECU) of the vehicle. Vehicle 200 may also beequipped with a position sensor 130, such as a GPS receiver and may alsoinclude a map database 160 and memory units 140 and 150.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle imaging system consistent with the disclosed embodiments. FIG.2B is a diagrammatic top view illustration of the embodiment shown inFIG. 2A. As illustrated in FIG. 2B, the disclosed embodiments mayinclude a vehicle 200 including in its body a system 100 with a firstimage capture device 122 positioned in the vicinity of the rearviewmirror and/or near the driver of vehicle 200, a second image capturedevice 124 positioned on or in a bumper region (e.g., one of bumperregions 210) of vehicle 200, and a processing unit 110.

As illustrated in FIG. 2C, image capture devices 122 and 124 may both bepositioned in the vicinity of the rearview mirror and/or near the driverof vehicle 200. Additionally, while two image capture devices 122 and124 are shown in FIGS. 2B and 2C, it should be understood that otherembodiments may include more than two image capture devices. Forexample, in the embodiments shown in FIGS. 2D and 2E, first, second, andthird image capture devices 122, 124, and 126, are included in thesystem 100 of vehicle 200.

As illustrated in FIG. 2D, image capture device 122 may be positioned inthe vicinity of the rearview mirror and/or near the driver of vehicle200, and image capture devices 124 and 126 may be positioned on or in abumper region (e.g., one of bumper regions 210) of vehicle 200. And asshown in FIG. 2E, image capture devices 122, 124, and 126 may bepositioned in the vicinity of the rearview mirror and/or near the driverseat of vehicle 200. The disclosed embodiments are not limited to anyparticular number and configuration of the image capture devices, andthe image capture devices may be positioned in any appropriate locationwithin and/or on vehicle 200.

It is to be understood that the disclosed embodiments are not limited tovehicles and could be applied in other contexts. It is also to beunderstood that disclosed embodiments are not limited to a particulartype of vehicle 200 and may be applicable to all types of vehiclesincluding automobiles, trucks, trailers, and other types of vehicles.

The first image capture device 122 may include any suitable type ofimage capture device. Image capture device 122 may include an opticalaxis. In one instance, the image capture device 122 may include anAptina M9V024 WVGA sensor with a global shutter. In other embodiments,image capture device 122 may provide a resolution of 1280×960 pixels andmay include a rolling shutter. Image capture device 122 may includevarious optical elements. In some embodiments one or more lenses may beincluded, for example, to provide a desired focal length and field ofview for the image capture device. In some embodiments, image capturedevice 122 may be associated with a 6 mm lens or a 12 mm lens. In someembodiments, image capture device 122 may be configured to captureimages having a desired field-of-view (FOV) 202, as illustrated in FIG.2D. For example, image capture device 122 may be configured to have aregular FOV, such as within a range of 40 degrees to 56 degrees,including a 46 degree FOV, 50 degree FOV, 52 degree FOV, or greater.Alternatively, image capture device 122 may be configured to have anarrow FOV in the range of 23 to 40 degrees, such as a 28 degree FOV or36 degree FOV. In addition, image capture device 122 may be configuredto have a wide FOV in the range of 100 to 180 degrees. In someembodiments, image capture device 122 may include a wide angle bumpercamera or one with up to a 180 degree FOV.

The first image capture device 122 may acquire a plurality of firstimages relative to a scene associated with the vehicle 200. Each of theplurality of first images may be acquired as a series of image scanlines, which may be captured using a rolling shutter. Each scan line mayinclude a plurality of pixels.

The first image capture device 122 may have a scan rate associated withacquisition of each of the first series of image scan lines. The scanrate may refer to a rate at which an image sensor can acquire image dataassociated with each pixel included in a particular scan line.

Image capture devices 122, 124, and 126 may contain any suitable typeand number of image sensors, including CCD sensors or CMOS sensors, forexample. In one embodiment, a CMOS image sensor may be employed alongwith a rolling shutter, such that each pixel in a row is read one at atime, and scanning of the rows proceeds on a row-by-row basis until anentire image frame has been captured. In some embodiments, the rows maybe captured sequentially from top to bottom relative to the frame.

The use of a rolling shutter may result in pixels in different rowsbeing exposed and captured at different times, which may cause skew andother image artifacts in the captured image frame. On the other hand,when the image capture device 122 is configured to operate with a globalor synchronous shutter, all of the pixels may be exposed for the sameamount of time and during a common exposure period. As a result, theimage data in a frame collected from a system employing a global shutterrepresents a snapshot of the entire FOV (such as FOV 202) at aparticular time. In contrast, in a rolling shutter application, each rowin a frame is exposed and data is capture at different times. Thus,moving objects may appear distorted in an image capture device having arolling shutter. This phenomenon will be described in greater detailbelow.

The second image capture device 124 and the third image capturing device126 may be any type of image capture device. Like the first imagecapture device 122, each of image capture devices 124 and 126 mayinclude an optical axis. In one embodiment, each of image capturedevices 124 and 126 may include an Aptina M9V024 WVGA sensor with aglobal shutter. Alternatively, each of image capture devices 124 and 126may include a rolling shutter. Like image capture device 122, imagecapture devices 124 and 126 may be configured to include various lensesand optical elements. In some embodiments, lenses associated with imagecapture devices 124 and 126 may provide FOVs (such as FOVs 204 and 206)that are the same as, or narrower than, a FOV (such as FOV 202)associated with image capture device 122. For example, image capturedevices 124 and 126 may have FOVs of 40 degrees, 30 degrees, 26 degrees,23 degrees, 20 degrees, or less.

Image capture devices 124 and 126 may acquire a plurality of second andthird images relative to a scene associated with the vehicle 200. Eachof the plurality of second and third images may be acquired as a secondand third series of image scan lines, which may be captured using arolling shutter. Each scan line or row may have a plurality of pixels.Image capture devices 124 and 126 may have second and third scan ratesassociated with acquisition of each of image scan lines included in thesecond and third series.

Each image capture device 122, 124, and 126 may be positioned at anysuitable position and orientation relative to vehicle 200. The relativepositioning of the image capture devices 122, 124, and 126 may beselected to aid in fusing together the information acquired from theimage capture devices. For example, in some embodiments, a FOV (such asFOV 204) associated with image capture device 124 may overlap partiallyor fully with a FOV (such as FOV 202) associated with image capturedevice 122 and a FOV (such as FOV 206) associated with image capturedevice 126.

Image capture devices 122, 124, and 126 may be located on vehicle 200 atany suitable relative heights. In one instance, there may be a heightdifference between the image capture devices 122, 124, and 126, whichmay provide sufficient parallax information to enable stereo analysis.For example, as shown in FIG. 2A, the two image capture devices 122 and124 are at different heights. There may also be a lateral displacementdifference between image capture devices 122, 124, and 126, givingadditional parallax information for stereo analysis by processing unit110, for example. The difference in the lateral displacement may bedenoted by d_(x), as shown in FIGS. 2C and 2D. In some embodiments, foreor aft displacement (e.g., range displacement) may exist between imagecapture devices 122, 124, and 126. For example, image capture device 122may be located 0.5 to 2 meters or more behind image capture device 124and/or image capture device 126. This type of displacement may enableone of the image capture devices to cover potential blind spots of theother image capture device(s).

Image capture devices 122 may have any suitable resolution capability(e.g., number of pixels associated with the image sensor), and theresolution of the image sensor(s) associated with the image capturedevice 122 may be higher, lower, or the same as the resolution of theimage sensor(s) associated with image capture devices 124 and 126. Insome embodiments, the image sensor(s) associated with image capturedevice 122 and/or image capture devices 124 and 126 may have aresolution of 640×480, 1024×768, 1280×960, or any other suitableresolution.

The frame rate (e.g., the rate at which an image capture device acquiresa set of pixel data of one image frame before moving on to capture pixeldata associated with the next image frame) may be controllable. Theframe rate associated with image capture device 122 may be higher,lower, or the same as the frame rate associated with image capturedevices 124 and 126. The frame rate associated with image capturedevices 122, 124, and 126 may depend on a variety of factors that mayaffect the timing of the frame rate. For example, one or more of imagecapture devices 122, 124, and 126 may include a selectable pixel delayperiod imposed before or after acquisition of image data associated withone or more pixels of an image sensor in image capture device 122, 124,and/or 126. Generally, image data corresponding to each pixel may beacquired according to a clock rate for the device (e.g., one pixel perclock cycle). Additionally, in embodiments including a rolling shutter,one or more of image capture devices 122, 124, and 126 may include aselectable horizontal blanking period imposed before or afteracquisition of image data associated with a row of pixels of an imagesensor in image capture device 122, 124, and/or 126. Further, one ormore of image capture devices 122, 124, and/or 126 may include aselectable vertical blanking period imposed before or after acquisitionof image data associated with an image frame of image capture device122, 124, and 126.

These timing controls may enable synchronization of frame ratesassociated with image capture devices 122, 124, and 126, even where theline scan rates of each are different. Additionally, as will bediscussed in greater detail below, these selectable timing controls,among other factors (e.g., image sensor resolution, maximum line scanrates, etc.) may enable synchronization of image capture from an areawhere the FOV of image capture device 122 overlaps with one or more FOVsof image capture devices 124 and 126, even where the field of view ofimage capture device 122 is different from the FOVs of image capturedevices 124 and 126.

Frame rate timing in image capture device 122, 124, and 126 may dependon the resolution of the associated image sensors. For example, assumingsimilar line scan rates for both devices, if one device includes animage sensor having a resolution of 640×480 and another device includesan image sensor with a resolution of 1280×960, then more time will berequired to acquire a frame of image data from the sensor having thehigher resolution.

Another factor that may affect the timing of image data acquisition inimage capture devices 122, 124, and 126 is the maximum line scan rate.For example, acquisition of a row of image data from an image sensorincluded in image capture device 122, 124, and 126 will require someminimum amount of time. Assuming no pixel delay periods are added, thisminimum amount of time for acquisition of a row of image data will berelated to the maximum line scan rate for a particular device. Devicesthat offer higher maximum line scan rates have the potential to providehigher frame rates than devices with lower maximum line scan rates. Insome embodiments, one or more of image capture devices 124 and 126 mayhave a maximum line scan rate that is higher than a maximum line scanrate associated with image capture device 122. In some embodiments, themaximum line scan rate of image capture device 124 and/or 126 may be1.25, 1.5, 1.75, or 2 times or more than a maximum line scan rate ofimage capture device 122.

In another embodiment, image capture devices 122, 124, and 126 may havethe same maximum line scan rate, but image capture device 122 may beoperated at a scan rate less than or equal to its maximum scan rate. Thesystem may be configured such that one or more of image capture devices124 and 126 operate at a line scan rate that is equal to the line scanrate of image capture device 122. In other instances, the system may beconfigured such that the line scan rate of image capture device 124and/or image capture device 126 may be 1.25, 1.5, 1.75, or 2 times ormore than the line scan rate of image capture device 122.

In some embodiments, image capture devices 122, 124, and 126 may beasymmetric. That is, they may include cameras having different fields ofview (FOV) and focal lengths. The fields of view of image capturedevices 122, 124, and 126 may include any desired area relative to anenvironment of vehicle 200, for example. In some embodiments, one ormore of image capture devices 122, 124, and 126 may be configured toacquire image data from an environment in front of vehicle 200, behindvehicle 200, to the sides of vehicle 200, or combinations thereof.

Further, the focal length associated with each image capture device 122,124, and/or 126 may be selectable (e.g., by inclusion of appropriatelenses etc.) such that each device acquires images of objects at adesired distance range relative to vehicle 200. For example, in someembodiments image capture devices 122, 124, and 126 may acquire imagesof close-up objects within a few meters from the vehicle. Image capturedevices 122, 124, and 126 may also be configured to acquire images ofobjects at ranges more distant from the vehicle (e.g., 25 m, 50 m, 100m, 150 m, or more). Further, the focal lengths of image capture devices122, 124, and 126 may be selected such that one image capture device(e.g., image capture device 122) can acquire images of objectsrelatively close to the vehicle (e.g., within 10 m or within 20 m) whilethe other image capture devices (e.g., image capture devices 124 and126) can acquire images of more distant objects (e.g., greater than 20m, 50 m, 100 m, 150 m, etc.) from vehicle 200.

According to some embodiments, the FOV of one or more image capturedevices 122, 124, and 126 may have a wide angle. For example, it may beadvantageous to have a FOV of 140 degrees, especially for image capturedevices 122, 124, and 126 that may be used to capture images of the areain the vicinity of vehicle 200. For example, image capture device 122may be used to capture images of the area to the right or left ofvehicle 200 and, in such embodiments, it may be desirable for imagecapture device 122 to have a wide FOV (e.g., at least 140 degrees).

The field of view associated with each of image capture devices 122,124, and 126 may depend on the respective focal lengths. For example, asthe focal length increases, the corresponding field of view decreases.

Image capture devices 122, 124, and 126 may be configured to have anysuitable fields of view. In one particular example, image capture device122 may have a horizontal FOV of 46 degrees, image capture device 124may have a horizontal FOV of 23 degrees, and image capture device 126may have a horizontal FOV in between 23 and 46 degrees. In anotherinstance, image capture device 122 may have a horizontal FOV of 52degrees, image capture device 124 may have a horizontal FOV of 26degrees, and image capture device 126 may have a horizontal FOV inbetween 26 and 52 degrees. In some embodiments, a ratio of the FOV ofimage capture device 122 to the FOVs of image capture device 124 and/orimage capture device 126 may vary from 1.5 to 2.0. In other embodiments,this ratio may vary between 1.25 and 2.25.

System 100 may be configured so that a field of view of image capturedevice 122 overlaps, at least partially or fully, with a field of viewof image capture device 124 and/or image capture device 126. In someembodiments, system 100 may be configured such that the fields of viewof image capture devices 124 and 126, for example, fall within (e.g.,are narrower than) and share a common center with the field of view ofimage capture device 122. In other embodiments, the image capturedevices 122, 124, and 126 may capture adjacent FOVs or may have partialoverlap in their FOVs. In some embodiments, the fields of view of imagecapture devices 122, 124, and 126 may be aligned such that a center ofthe narrower FOV image capture devices 124 and/or 126 may be located ina lower half of the field of view of the wider FOV device 122.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems, consistent with the disclosed embodiments. As indicated in FIG.2F, vehicle 200 may include throttling system 220, braking system 230,and steering system 240. System 100 may provide inputs (e.g., controlsignals) to one or more of throttling system 220, braking system 230,and steering system 240 over one or more data links (e.g., any wiredand/or wireless link or links for transmitting data). For example, basedon analysis of images acquired by image capture devices 122, 124, and/or126, system 100 may provide control signals to one or more of throttlingsystem 220, braking system 230, and steering system 240 to navigatevehicle 200 (e.g., by causing an acceleration, a turn, a lane shift,etc.). Further, system 100 may receive inputs from one or more ofthrottling system 220, braking system 230, and steering system 24indicating operating conditions of vehicle 200 (e.g., speed, whethervehicle 200 is braking and/or turning, etc.). Further details areprovided in connection with FIGS. 4-7 , below.

As shown in FIG. 3A, vehicle 200 may also include a user interface 170for interacting with a driver or a passenger of vehicle 200. Forexample, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver orpassenger of vehicle 200 may also use handles (e.g., located on or nearthe steering column of vehicle 200 including, for example, turn signalhandles), buttons (e.g., located on the steering wheel of vehicle 200),and the like, to interact with system 100. In some embodiments,microphone 350 may be positioned adjacent to a rearview mirror 310.Similarly, in some embodiments, image capture device 122 may be locatednear rearview mirror 310. In some embodiments, user interface 170 mayalso include one or more speakers 360 (e.g., speakers of a vehicle audiosystem). For example, system 100 may provide various notifications(e.g., alerts) via speakers 360.

FIGS. 3B-3D are illustrations of an exemplary camera mount 370configured to be positioned behind a rearview mirror (e.g., rearviewmirror 310) and against a vehicle windshield, consistent with disclosedembodiments. As shown in FIG. 3B, camera mount 370 may include imagecapture devices 122, 124, and 126. Image capture devices 124 and 126 maybe positioned behind a glare shield 380, which may be flush against thevehicle windshield and include a composition of film and/oranti-reflective materials. For example, glare shield 380 may bepositioned such that it aligns against a vehicle windshield having amatching slope. In some embodiments, each of image capture devices 122,124, and 126 may be positioned behind glare shield 380, as depicted, forexample, in FIG. 3D. The disclosed embodiments are not limited to anyparticular configuration of image capture devices 122, 124, and 126,camera mount 370, and glare shield 380. FIG. 3C is an illustration ofcamera mount 370 shown in FIG. 3B from a front perspective.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the foregoing disclosed embodiments. For example, not all componentsare essential for the operation of system 100. Further, any componentmay be located in any appropriate part of system 100 and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. Therefore, the foregoingconfigurations are examples and, regardless of the configurationsdiscussed above, system 100 can provide a wide range of functionality toanalyze the surroundings of vehicle 200 and navigate vehicle 200 inresponse to the analysis.

As discussed below in further detail and consistent with variousdisclosed embodiments, system 100 may provide a variety of featuresrelated to autonomous driving and/or driver assist technology. Forexample, system 100 may analyze image data, position data (e.g., GPSlocation information), map data, speed data, and/or data from sensorsincluded in vehicle 200. System 100 may collect the data for analysisfrom, for example, image acquisition unit 120, position sensor 130, andother sensors. Further, system 100 may analyze the collected data todetermine whether or not vehicle 200 should take a certain action, andthen automatically take the determined action without humanintervention. For example, when vehicle 200 navigates without humanintervention, system 100 may automatically control the braking,acceleration, and/or steering of vehicle 200 (e.g., by sending controlsignals to one or more of throttling system 220, braking system 230, andsteering system 240). Further, system 100 may analyze the collected dataand issue warnings and/or alerts to vehicle occupants based on theanalysis of the collected data.

Further, consistent with disclosed embodiments, the functionalityprovided by system 100 may cause vehicle 200 to take different actionsto navigate vehicle 200 within a lane and/or relative to other vehiclesand/or objects. For example, system 100 may adjust the positioning ofvehicle 200 relative to a lane within which vehicle 200 is travelingand/or relative to objects positioned near vehicle 200, select aparticular lane for vehicle 200 to use while traveling, and take actionin response to an encroaching vehicle, such as a vehicle attempting tomove into the lane within which vehicle 200 is traveling. Additionally,system 100 may control the speed of vehicle 200 in different scenarios,such as when vehicle 200 is making a turn. System 100 may further causevehicle 200 to mimic the actions of a leading vehicle or monitor atarget vehicle and navigate vehicle 200 so that it passes the targetvehicle. Additional details regarding the various embodiments that areprovided by system 100 are provided below.

Forward-Facing Multi-Imaging System

As discussed above, system 100 may provide drive assist functionalitythat uses a multi-camera system. The multi-camera system may use one ormore cameras facing in the forward direction of a vehicle. In otherembodiments, the multi-camera system may include one or more camerasfacing to the side of a vehicle or to the rear of the vehicle. In oneembodiment, for example, system 100 may use a two-camera imaging system,where a first camera and a second camera (e.g., image capture devices122 and 124) may be positioned at the front and/or the sides of avehicle (e.g., vehicle 200). The first camera may have a field of viewthat is greater than, less than, or partially overlapping with, thefield of view of the second camera. In addition, the first camera may beconnected to a first image processor to perform monocular image analysisof images provided by the first camera, and the second camera may beconnected to a second image processor to perform monocular imageanalysis of images provided by the second camera. The outputs (e.g.,processed information) of the first and second image processors may becombined. In some embodiments, the second image processor may receiveimages from both the first camera and second camera to perform stereoanalysis. In another embodiment, system 100 may use a three-cameraimaging system where each of the cameras has a different field of view.Such a system may, therefore, make decisions based on informationderived from objects located at varying distances both forward and tothe sides of the vehicle. References to monocular image analysis mayrefer to instances where image analysis is performed based on imagescaptured from a single point of view (e.g., from a single camera).Stereo image analysis may refer to instances where image analysis isperformed based on two or more images captured with one or morevariations of an image capture parameter. For example, captured imagessuitable for performing stereo image analysis may include imagescaptured: from two or more different positions, from different fields ofview, using different focal lengths, along with parallax information,etc.

For example, in one embodiment, system 100 may implement a three cameraconfiguration using image capture devices 122-126. In such aconfiguration, image capture device 122 may provide a narrow field ofview (e.g., 34 degrees, or other values selected from a range of about20 to 45 degrees, etc.), image capture device 124 may provide a widefield of view (e.g., 150 degrees or other values selected from a rangeof about 100 to about 180 degrees), and image capture device 126 mayprovide an intermediate field of view (e.g., 46 degrees or other valuesselected from a range of about 35 to about 60 degrees). In someembodiments, image capture device 126 may act as a main or primarycamera. Image capture devices 122-126 may be positioned behind rearviewmirror 310 and positioned substantially side-by-side (e.g., 6 cm apart).Further, in some embodiments, as discussed above, one or more of imagecapture devices 122-126 may be mounted behind glare shield 380 that isflush with the windshield of vehicle 200. Such shielding may act tominimize the impact of any reflections from inside the car on imagecapture devices 122-126.

In another embodiment, as discussed above in connection with FIGS. 3Band 3C, the wide field of view camera (e.g., image capture device 124 inthe above example) may be mounted lower than the narrow and main fieldof view cameras (e.g., image devices 122 and 126 in the above example).This configuration may provide a free line of sight from the wide fieldof view camera. To reduce reflections, the cameras may be mounted closeto the windshield of vehicle 200, and may include polarizers on thecameras to damp reflected light.

A three camera system may provide certain performance characteristics.For example, some embodiments may include an ability to validate thedetection of objects by one camera based on detection results fromanother camera. In the three camera configuration discussed above,processing unit 110 may include, for example, three processing devices(e.g., three EyeQ series of processor chips, as discussed above), witheach processing device dedicated to processing images captured by one ormore of image capture devices 122-126.

In a three camera system, a first processing device may receive imagesfrom both the main camera and the narrow field of view camera, andperform vision processing of the narrow FOV camera to, for example,detect other vehicles, pedestrians, lane marks, traffic signs, trafficlights, and other road objects. Further, the first processing device maycalculate a disparity of pixels between the images from the main cameraand the narrow camera and create a 3D reconstruction of the environmentof vehicle 200. The first processing device may then combine the 3Dreconstruction with 3D map data or with 3D information calculated basedon information from another camera.

The second processing device may receive images from main camera andperform vision processing to detect other vehicles, pedestrians, lanemarks, traffic signs, traffic lights, and other road objects.Additionally, the second processing device may calculate a cameradisplacement and, based on the displacement, calculate a disparity ofpixels between successive images and create a 3D reconstruction of thescene (e.g., a structure from motion). The second processing device maysend the structure from motion based 3D reconstruction to the firstprocessing device to be combined with the stereo 3D images.

The third processing device may receive images from the wide FOV cameraand process the images to detect vehicles, pedestrians, lane marks,traffic signs, traffic lights, and other road objects. The thirdprocessing device may further execute additional processing instructionsto analyze images to identify objects moving in the image, such asvehicles changing lanes, pedestrians, etc.

In some embodiments, having streams of image-based information capturedand processed independently may provide an opportunity for providingredundancy in the system. Such redundancy may include, for example,using a first image capture device and the images processed from thatdevice to validate and/or supplement information obtained by capturingand processing image information from at least a second image capturedevice.

In some embodiments, system 100 may use two image capture devices (e.g.,image capture devices 122 and 124) in providing navigation assistancefor vehicle 200 and use a third image capture device (e.g., imagecapture device 126) to provide redundancy and validate the analysis ofdata received from the other two image capture devices. For example, insuch a configuration, image capture devices 122 and 124 may provideimages for stereo analysis by system 100 for navigating vehicle 200,while image capture device 126 may provide images for monocular analysisby system 100 to provide redundancy and validation of informationobtained based on images captured from image capture device 122 and/orimage capture device 124. That is, image capture device 126 (and acorresponding processing device) may be considered to provide aredundant sub-system for providing a check on the analysis derived fromimage capture devices 122 and 124 (e.g., to provide an automaticemergency braking (AEB) system).

One of skill in the art will recognize that the above cameraconfigurations, camera placements, number of cameras, camera locations,etc., are examples only. These components and others described relativeto the overall system may be assembled and used in a variety ofdifferent configurations without departing from the scope of thedisclosed embodiments. Further details regarding usage of a multi-camerasystem to provide driver assist and/or autonomous vehicle functionalityfollow below.

FIG. 4 is an exemplary functional block diagram of memory 140 and/or150, which may be stored/programmed with instructions for performing oneor more operations consistent with the disclosed embodiments. Althoughthe following refers to memory 140, one of skill in the art willrecognize that instructions may be stored in memory 140 and/or 150.

As shown in FIG. 4 , memory 140 may store a monocular image analysismodule 402, a stereo image analysis module 404, a velocity andacceleration module 406, and a navigational response module 408. Thedisclosed embodiments are not limited to any particular configuration ofmemory 140. Further, application processor 180 and/or image processor190 may execute the instructions stored in any of modules 402-408included in memory 140. One of skill in the art will understand thatreferences in the following discussions to processing unit 110 may referto application processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

In one embodiment, monocular image analysis module 402 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs monocular image analysis of a set ofimages acquired by one of image capture devices 122, 124, and 126. Insome embodiments, processing unit 110 may combine information from a setof images with additional sensory information (e.g., information fromradar) to perform the monocular image analysis. As described inconnection with FIGS. 5A-5D below, monocular image analysis module 402may include instructions for detecting a set of features within the setof images, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, hazardous objects, and any otherfeature associated with an environment of a vehicle. Based on theanalysis, system 100 (e.g., via processing unit 110) may cause one ormore navigational responses in vehicle 200, such as a turn, a laneshift, a change in acceleration, and the like, as discussed below inconnection with navigational response module 408.

In one embodiment, stereo image analysis module 404 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs stereo image analysis of first and secondsets of images acquired by a combination of image capture devicesselected from any of image capture devices 122, 124, and 126. In someembodiments, processing unit 110 may combine information from the firstand second sets of images with additional sensory information (e.g.,information from radar) to perform the stereo image analysis. Forexample, stereo image analysis module 404 may include instructions forperforming stereo image analysis based on a first set of images acquiredby image capture device 124 and a second set of images acquired by imagecapture device 126. As described in connection with FIG. 6 below, stereoimage analysis module 404 may include instructions for detecting a setof features within the first and second sets of images, such as lanemarkings, vehicles, pedestrians, road signs, highway exit ramps, trafficlights, hazardous objects, and the like. Based on the analysis,processing unit 110 may cause one or more navigational responses invehicle 200, such as a turn, a lane shift, a change in acceleration, andthe like, as discussed below in connection with navigational responsemodule 408.

In one embodiment, velocity and acceleration module 406 may storesoftware configured to analyze data received from one or more computingand electromechanical devices in vehicle 200 that are configured tocause a change in velocity and/or acceleration of vehicle 200. Forexample, processing unit 110 may execute instructions associated withvelocity and acceleration module 406 to calculate a target speed forvehicle 200 based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include, for example, a target position, velocity, and/oracceleration, the position and/or speed of vehicle 200 relative to anearby vehicle, pedestrian, or road object, position information forvehicle 200 relative to lane markings of the road, and the like. Inaddition, processing unit 110 may calculate a target speed for vehicle200 based on sensory input (e.g., information from radar) and input fromother systems of vehicle 200, such as throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200. Based on thecalculated target speed, processing unit 110 may transmit electronicsignals to throttling system 220, braking system 230, and/or steeringsystem 240 of vehicle 200 to trigger a change in velocity and/oracceleration by, for example, physically depressing the brake or easingup off the accelerator of vehicle 200.

In one embodiment, navigational response module 408 may store softwareexecutable by processing unit 110 to determine a desired navigationalresponse based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include position and speed information associated with nearbyvehicles, pedestrians, and road objects, target position information forvehicle 200, and the like. Additionally, in some embodiments, thenavigational response may be based (partially or fully) on map data, apredetermined position of vehicle 200, and/or a relative velocity or arelative acceleration between vehicle 200 and one or more objectsdetected from execution of monocular image analysis module 402 and/orstereo image analysis module 404. Navigational response module 408 mayalso determine a desired navigational response based on sensory input(e.g., information from radar) and inputs from other systems of vehicle200, such as throttling system 220, braking system 230, and steeringsystem 240 of vehicle 200. Based on the desired navigational response,processing unit 110 may transmit electronic signals to throttling system220, braking system 230, and steering system 240 of vehicle 200 totrigger a desired navigational response by, for example, turning thesteering wheel of vehicle 200 to achieve a rotation of a predeterminedangle. In some embodiments, processing unit 110 may use the output ofnavigational response module 408 (e.g., the desired navigationalresponse) as an input to execution of velocity and acceleration module406 for calculating a change in speed of vehicle 200.

FIG. 5A is a flowchart showing an exemplary process 500A for causing oneor more navigational responses based on monocular image analysis,consistent with disclosed embodiments. At step 510, processing unit 110may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120. For instance, acamera included in image acquisition unit 120 (such as image capturedevice 122 having field of view 202) may capture a plurality of imagesof an area forward of vehicle 200 (or to the sides or rear of a vehicle,for example) and transmit them over a data connection (e.g., digital,wired, USB, wireless, Bluetooth, etc.) to processing unit 110.Processing unit 110 may execute monocular image analysis module 402 toanalyze the plurality of images at step 520, as described in furtherdetail in connection with FIGS. 5B-5D below. By performing the analysis,processing unit 110 may detect a set of features within the set ofimages, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, and the like.

Processing unit 110 may also execute monocular image analysis module 402to detect various road hazards at step 520, such as, for example, partsof a truck tire, fallen road signs, loose cargo, small animals, and thelike. Road hazards may vary in structure, shape, size, and color, whichmay make detection of such hazards more challenging. In someembodiments, processing unit 110 may execute monocular image analysismodule 402 to perform multi-frame analysis on the plurality of images todetect road hazards. For example, processing unit 110 may estimatecamera motion between consecutive image frames and calculate thedisparities in pixels between the frames to construct a 3D-map of theroad. Processing unit 110 may then use the 3D-map to detect the roadsurface, as well as hazards existing above the road surface.

At step 530, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 520 and the techniques asdescribed above in connection with FIG. 4 . Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration,and the like. In some embodiments, processing unit 110 may use dataderived from execution of velocity and acceleration module 406 to causethe one or more navigational responses. Additionally, multiplenavigational responses may occur simultaneously, in sequence, or anycombination thereof. For instance, processing unit 110 may cause vehicle200 to shift one lane over and then accelerate by, for example,sequentially transmitting control signals to steering system 240 andthrottling system 220 of vehicle 200. Alternatively, processing unit 110may cause vehicle 200 to brake while at the same time shifting lanes by,for example, simultaneously transmitting control signals to brakingsystem 230 and steering system 240 of vehicle 200.

FIG. 5B is a flowchart showing an exemplary process 500B for detectingone or more vehicles and/or pedestrians in a set of images, consistentwith disclosed embodiments. Processing unit 110 may execute monocularimage analysis module 402 to implement process 500B. At step 540,processing unit 110 may determine a set of candidate objectsrepresenting possible vehicles and/or pedestrians. For example,processing unit 110 may scan one or more images, compare the images toone or more predetermined patterns, and identify within each imagepossible locations that may contain objects of interest (e.g., vehicles,pedestrians, or portions thereof). The predetermined patterns may bedesigned in such a way to achieve a high rate of “false hits” and a lowrate of “misses.” For example, processing unit 110 may use a lowthreshold of similarity to predetermined patterns for identifyingcandidate objects as possible vehicles or pedestrians. Doing so mayallow processing unit 110 to reduce the probability of missing (e.g.,not identifying) a candidate object representing a vehicle orpedestrian.

At step 542, processing unit 110 may filter the set of candidate objectsto exclude certain candidates (e.g., irrelevant or less relevantobjects) based on classification criteria. Such criteria may be derivedfrom various properties associated with object types stored in adatabase (e.g., a database stored in memory 140). Properties may includeobject shape, dimensions, texture, position (e.g., relative to vehicle200), and the like. Thus, processing unit 110 may use one or more setsof criteria to reject false candidates from the set of candidateobjects.

At step 544, processing unit 110 may analyze multiple frames of imagesto determine whether objects in the set of candidate objects representvehicles and/or pedestrians. For example, processing unit 110 may tracka detected candidate object across consecutive frames and accumulateframe-by-frame data associated with the detected object (e.g., size,position relative to vehicle 200, etc.). Additionally, processing unit110 may estimate parameters for the detected object and compare theobject's frame-by-frame position data to a predicted position.

At step 546, processing unit 110 may construct a set of measurements forthe detected objects. Such measurements may include, for example,position, velocity, and acceleration values (relative to vehicle 200)associated with the detected objects. In some embodiments, processingunit 110 may construct the measurements based on estimation techniquesusing a series of time-based observations such as Kalman filters orlinear quadratic estimation (LQE), and/or based on available modelingdata for different object types (e.g., cars, trucks, pedestrians,bicycles, road signs, etc.). The Kalman filters may be based on ameasurement of an object's scale, where the scale measurement isproportional to a time to collision (e.g., the amount of time forvehicle 200 to reach the object). Thus, by performing steps 540-546,processing unit 110 may identify vehicles and pedestrians appearingwithin the set of captured images and derive information (e.g.,position, speed, size) associated with the vehicles and pedestrians.Based on the identification and the derived information, processing unit110 may cause one or more navigational responses in vehicle 200, asdescribed in connection with FIG. 5A, above.

At step 548, processing unit 110 may perform an optical flow analysis ofone or more images to reduce the probabilities of detecting a “falsehit” and missing a candidate object that represents a vehicle orpedestrian. The optical flow analysis may refer to, for example,analyzing motion patterns relative to vehicle 200 in the one or moreimages associated with other vehicles and pedestrians, and that aredistinct from road surface motion. Processing unit 110 may calculate themotion of candidate objects by observing the different positions of theobjects across multiple image frames, which are captured at differenttimes. Processing unit 110 may use the position and time values asinputs into mathematical models for calculating the motion of thecandidate objects. Thus, optical flow analysis may provide anothermethod of detecting vehicles and pedestrians that are nearby vehicle200. Processing unit 110 may perform optical flow analysis incombination with steps 540-546 to provide redundancy for detectingvehicles and pedestrians and increase the reliability of system 100.

FIG. 5C is a flowchart showing an exemplary process 500C for detectingroad marks and/or lane geometry information in a set of images,consistent with disclosed embodiments. Processing unit 110 may executemonocular image analysis module 402 to implement process 500C. At step550, processing unit 110 may detect a set of objects by scanning one ormore images. To detect segments of lane markings, lane geometryinformation, and other pertinent road marks, processing unit 110 mayfilter the set of objects to exclude those determined to be irrelevant(e.g., minor potholes, small rocks, etc.). At step 552, processing unit110 may group together the segments detected in step 550 belonging tothe same road mark or lane mark. Based on the grouping, processing unit110 may develop a model to represent the detected segments, such as amathematical model.

At step 554, processing unit 110 may construct a set of measurementsassociated with the detected segments. In some embodiments, processingunit 110 may create a projection of the detected segments from the imageplane onto the real-world plane. The projection may be characterizedusing a 3rd-degree polynomial having coefficients corresponding tophysical properties such as the position, slope, curvature, andcurvature derivative of the detected road. In generating the projection,processing unit 110 may take into account changes in the road surface,as well as pitch and roll rates associated with vehicle 200. Inaddition, processing unit 110 may model the road elevation by analyzingposition and motion cues present on the road surface. Further,processing unit 110 may estimate the pitch and roll rates associatedwith vehicle 200 by tracking a set of feature points in the one or moreimages.

At step 556, processing unit 110 may perform multi-frame analysis by,for example, tracking the detected segments across consecutive imageframes and accumulating frame-by-frame data associated with detectedsegments. As processing unit 110 performs multi-frame analysis, the setof measurements constructed at step 554 may become more reliable andassociated with an increasingly higher confidence level. Thus, byperforming steps 550-556, processing unit 110 may identify road marksappearing within the set of captured images and derive lane geometryinformation. Based on the identification and the derived information,processing unit 110 may cause one or more navigational responses invehicle 200, as described in connection with FIG. 5A, above.

At step 558, processing unit 110 may consider additional sources ofinformation to further develop a safety model for vehicle 200 in thecontext of its surroundings. Processing unit 110 may use the safetymodel to define a context in which system 100 may execute autonomouscontrol of vehicle 200 in a safe manner. To develop the safety model, insome embodiments, processing unit 110 may consider the position andmotion of other vehicles, the detected road edges and barriers, and/orgeneral road shape descriptions extracted from map data (such as datafrom map database 160). By considering additional sources ofinformation, processing unit 110 may provide redundancy for detectingroad marks and lane geometry and increase the reliability of system 100.

FIG. 5D is a flowchart showing an exemplary process 500D for detectingtraffic lights in a set of images, consistent with disclosedembodiments. Processing unit 110 may execute monocular image analysismodule 402 to implement process 500D. At step 560, processing unit 110may scan the set of images and identify objects appearing at locationsin the images likely to contain traffic lights. For example, processingunit 110 may filter the identified objects to construct a set ofcandidate objects, excluding those objects unlikely to correspond totraffic lights. The filtering may be done based on various propertiesassociated with traffic lights, such as shape, dimensions, texture,position (e.g., relative to vehicle 200), and the like. Such propertiesmay be based on multiple examples of traffic lights and traffic controlsignals and stored in a database. In some embodiments, processing unit110 may perform multi-frame analysis on the set of candidate objectsreflecting possible traffic lights. For example, processing unit 110 maytrack the candidate objects across consecutive image frames, estimatethe real-world position of the candidate objects, and filter out thoseobjects that are moving (which are unlikely to be traffic lights). Insome embodiments, processing unit 110 may perform color analysis on thecandidate objects and identify the relative position of the detectedcolors appearing inside possible traffic lights.

At step 562, processing unit 110 may analyze the geometry of a junction.The analysis may be based on any combination of: (i) the number of lanesdetected on either side of vehicle 200, (ii) markings (such as arrowmarks) detected on the road, and (iii) descriptions of the junctionextracted from map data (such as data from map database 160). Processingunit 110 may conduct the analysis using information derived fromexecution of monocular analysis module 402. In addition, Processing unit110 may determine a correspondence between the traffic lights detectedat step 560 and the lanes appearing near vehicle 200.

As vehicle 200 approaches the junction, at step 564, processing unit 110may update the confidence level associated with the analyzed junctiongeometry and the detected traffic lights. For instance, the number oftraffic lights estimated to appear at the junction as compared with thenumber actually appearing at the junction may impact the confidencelevel. Thus, based on the confidence level, processing unit 110 maydelegate control to the driver of vehicle 200 in order to improve safetyconditions. By performing steps 560-564, processing unit 110 mayidentify traffic lights appearing within the set of captured images andanalyze junction geometry information. Based on the identification andthe analysis, processing unit 110 may cause one or more navigationalresponses in vehicle 200, as described in connection with FIG. 5A,above.

FIG. 5E is a flowchart showing an exemplary process 500E for causing oneor more navigational responses in vehicle 200 based on a vehicle path,consistent with the disclosed embodiments. At step 570, processing unit110 may construct an initial vehicle path associated with vehicle 200.The vehicle path may be represented using a set of points expressed incoordinates (x, z), and the distance d_(i) between two points in the setof points may fall in the range of 1 to 5 meters. In one embodiment,processing unit 110 may construct the initial vehicle path using twopolynomials, such as left and right road polynomials. Processing unit110 may calculate the geometric midpoint between the two polynomials andoffset each point included in the resultant vehicle path by apredetermined offset (e.g., a smart lane offset), if any (an offset ofzero may correspond to travel in the middle of a lane). The offset maybe in a direction perpendicular to a segment between any two points inthe vehicle path. In another embodiment, processing unit 110 may use onepolynomial and an estimated lane width to offset each point of thevehicle path by half the estimated lane width plus a predeterminedoffset (e.g., a smart lane offset).

At step 572, processing unit 110 may update the vehicle path constructedat step 570. Processing unit 110 may reconstruct the vehicle pathconstructed at step 570 using a higher resolution, such that thedistance d_(k) between two points in the set of points representing thevehicle path is less than the distance d_(i) described above. Forexample, the distance d_(k) may fall in the range of 0.1 to 0.3 meters.Processing unit 110 may reconstruct the vehicle path using a parabolicspline algorithm, which may yield a cumulative distance vector Scorresponding to the total length of the vehicle path (i.e., based onthe set of points representing the vehicle path).

At step 574, processing unit 110 may determine a look-ahead point(expressed in coordinates as (x_(l), z_(l))) based on the updatedvehicle path constructed at step 572. Processing unit 110 may extractthe look-ahead point from the cumulative distance vector S, and thelook-ahead point may be associated with a look-ahead distance andlook-ahead time. The look-ahead distance, which may have a lower boundranging from 10 to 20 meters, may be calculated as the product of thespeed of vehicle 200 and the look-ahead time. For example, as the speedof vehicle 200 decreases, the look-ahead distance may also decrease(e.g., until it reaches the lower bound). The look-ahead time, which mayrange from 0.5 to 1.5 seconds, may be inversely proportional to the gainof one or more control loops associated with causing a navigationalresponse in vehicle 200, such as the heading error tracking controlloop. For example, the gain of the heading error tracking control loopmay depend on the bandwidth of a yaw rate loop, a steering actuatorloop, car lateral dynamics, and the like. Thus, the higher the gain ofthe heading error tracking control loop, the lower the look-ahead time.

At step 576, processing unit 110 may determine a heading error and yawrate command based on the look-ahead point determined at step 574.Processing unit 110 may determine the heading error by calculating thearctangent of the look-ahead point, e.g., arctan (x_(l)/z_(l)).Processing unit 110 may determine the yaw rate command as the product ofthe heading error and a high-level control gain. The high-level controlgain may be equal to: (2/look-ahead time), if the look-ahead distance isnot at the lower bound. Otherwise, the high-level control gain may beequal to: (2*speed of vehicle 200/look-ahead distance).

FIG. 5F is a flowchart showing an exemplary process 500F for determiningwhether a leading vehicle is changing lanes, consistent with thedisclosed embodiments. At step 580, processing unit 110 may determinenavigation information associated with a leading vehicle (e.g., avehicle traveling ahead of vehicle 200). For example, processing unit110 may determine the position, velocity (e.g., direction and speed),and/or acceleration of the leading vehicle, using the techniquesdescribed in connection with FIGS. 5A and 5B, above. Processing unit 110may also determine one or more road polynomials, a look-ahead point(associated with vehicle 200), and/or a snail trail (e.g., a set ofpoints describing a path taken by the leading vehicle), using thetechniques described in connection with FIG. 5E, above.

At step 582, processing unit 110 may analyze the navigation informationdetermined at step 580. In one embodiment, processing unit 110 maycalculate the distance between a snail trail and a road polynomial(e.g., along the trail). If the variance of this distance along thetrail exceeds a predetermined threshold (for example, 0.1 to 0.2 meterson a straight road, 0.3 to 0.4 meters on a moderately curvy road, and0.5 to 0.6 meters on a road with sharp curves), processing unit 110 maydetermine that the leading vehicle is likely changing lanes. In the casewhere multiple vehicles are detected traveling ahead of vehicle 200,processing unit 110 may compare the snail trails associated with eachvehicle. Based on the comparison, processing unit 110 may determine thata vehicle whose snail trail does not match with the snail trails of theother vehicles is likely changing lanes. Processing unit 110 mayadditionally compare the curvature of the snail trail (associated withthe leading vehicle) with the expected curvature of the road segment inwhich the leading vehicle is traveling. The expected curvature may beextracted from map data (e.g., data from map database 160), from roadpolynomials, from other vehicles' snail trails, from prior knowledgeabout the road, and the like. If the difference in curvature of thesnail trail and the expected curvature of the road segment exceeds apredetermined threshold, processing unit 110 may determine that theleading vehicle is likely changing lanes.

In another embodiment, processing unit 110 may compare the leadingvehicle's instantaneous position with the look-ahead point (associatedwith vehicle 200) over a specific period of time (e.g., 0.5 to 1.5seconds). If the distance between the leading vehicle's instantaneousposition and the look-ahead point varies during the specific period oftime, and the cumulative sum of variation exceeds a predeterminedthreshold (for example, 0.3 to 0.4 meters on a straight road, 0.7 to 0.8meters on a moderately curvy road, and 1.3 to 1.7 meters on a road withsharp curves), processing unit 110 may determine that the leadingvehicle is likely changing lanes. In another embodiment, processing unit110 may analyze the geometry of the snail trail by comparing the lateraldistance traveled along the trail with the expected curvature of thesnail trail. The expected radius of curvature may be determinedaccording to the calculation: (δ_(z) ²+δ_(x) ²)/2/(δ_(x)), where δ_(x)represents the lateral distance traveled and δ_(z) represents thelongitudinal distance traveled. If the difference between the lateraldistance traveled and the expected curvature exceeds a predeterminedthreshold (e.g., 500 to 700 meters), processing unit 110 may determinethat the leading vehicle is likely changing lanes. In anotherembodiment, processing unit 110 may analyze the position of the leadingvehicle. If the position of the leading vehicle obscures a roadpolynomial (e.g., the leading vehicle is overlaid on top of the roadpolynomial), then processing unit 110 may determine that the leadingvehicle is likely changing lanes. In the case where the position of theleading vehicle is such that, another vehicle is detected ahead of theleading vehicle and the snail trails of the two vehicles are notparallel, processing unit 110 may determine that the (closer) leadingvehicle is likely changing lanes.

At step 584, processing unit 110 may determine whether or not leadingvehicle 200 is changing lanes based on the analysis performed at step582. For example, processing unit 110 may make the determination basedon a weighted average of the individual analyses performed at step 582.Under such a scheme, for example, a decision by processing unit 110 thatthe leading vehicle is likely changing lanes based on a particular typeof analysis may be assigned a value of “1” (and “0” to represent adetermination that the leading vehicle is not likely changing lanes).Different analyses performed at step 582 may be assigned differentweights, and the disclosed embodiments are not limited to any particularcombination of analyses and weights.

FIG. 6 is a flowchart showing an exemplary process 600 for causing oneor more navigational responses based on stereo image analysis,consistent with disclosed embodiments. At step 610, processing unit 110may receive a first and second plurality of images via data interface128. For example, cameras included in image acquisition unit 120 (suchas image capture devices 122 and 124 having fields of view 202 and 204)may capture a first and second plurality of images of an area forward ofvehicle 200 and transmit them over a digital connection (e.g., USB,wireless, Bluetooth, etc.) to processing unit 110. In some embodiments,processing unit 110 may receive the first and second plurality of imagesvia two or more data interfaces. The disclosed embodiments are notlimited to any particular data interface configurations or protocols.

At step 620, processing unit 110 may execute stereo image analysismodule 404 to perform stereo image analysis of the first and secondplurality of images to create a 3D map of the road in front of thevehicle and detect features within the images, such as lane markings,vehicles, pedestrians, road signs, highway exit ramps, traffic lights,road hazards, and the like. Stereo image analysis may be performed in amanner similar to the steps described in connection with FIGS. 5A-5D,above. For example, processing unit 110 may execute stereo imageanalysis module 404 to detect candidate objects (e.g., vehicles,pedestrians, road marks, traffic lights, road hazards, etc.) within thefirst and second plurality of images, filter out a subset of thecandidate objects based on various criteria, and perform multi-frameanalysis, construct measurements, and determine a confidence level forthe remaining candidate objects. In performing the steps above,processing unit 110 may consider information from both the first andsecond plurality of images, rather than information from one set ofimages alone. For example, processing unit 110 may analyze thedifferences in pixel-level data (or other data subsets from among thetwo streams of captured images) for a candidate object appearing in boththe first and second plurality of images. As another example, processingunit 110 may estimate a position and/or velocity of a candidate object(e.g., relative to vehicle 200) by observing that the object appears inone of the plurality of images but not the other or relative to otherdifferences that may exist relative to objects appearing if the twoimage streams. For example, position, velocity, and/or accelerationrelative to vehicle 200 may be determined based on trajectories,positions, movement characteristics, etc. of features associated with anobject appearing in one or both of the image streams.

At step 630, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 620 and the techniques asdescribed above in connection with FIG. 4 . Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration, achange in velocity, braking, and the like. In some embodiments,processing unit 110 may use data derived from execution of velocity andacceleration module 406 to cause the one or more navigational responses.Additionally, multiple navigational responses may occur simultaneously,in sequence, or any combination thereof.

FIG. 7 is a flowchart showing an exemplary process 700 for causing oneor more navigational responses based on an analysis of three sets ofimages, consistent with disclosed embodiments. At step 710, processingunit 110 may receive a first, second, and third plurality of images viadata interface 128. For instance, cameras included in image acquisitionunit 120 (such as image capture devices 122, 124, and 126 having fieldsof view 202, 204, and 206) may capture a first, second, and thirdplurality of images of an area forward and/or to the side of vehicle 200and transmit them over a digital connection (e.g., USB, wireless,Bluetooth, etc.) to processing unit 110. In some embodiments, processingunit 110 may receive the first, second, and third plurality of imagesvia three or more data interfaces. For example, each of image capturedevices 122, 124, 126 may have an associated data interface forcommunicating data to processing unit 110. The disclosed embodiments arenot limited to any particular data interface configurations orprotocols.

At step 720, processing unit 110 may analyze the first, second, andthird plurality of images to detect features within the images, such aslane markings, vehicles, pedestrians, road signs, highway exit ramps,traffic lights, road hazards, and the like. The analysis may beperformed in a manner similar to the steps described in connection withFIGS. 5A-5D and 6 , above. For instance, processing unit 110 may performmonocular image analysis (e.g., via execution of monocular imageanalysis module 402 and based on the steps described in connection withFIGS. 5A-5D, above) on each of the first, second, and third plurality ofimages. Alternatively, processing unit 110 may perform stereo imageanalysis (e.g., via execution of stereo image analysis module 404 andbased on the steps described in connection with FIG. 6 , above) on thefirst and second plurality of images, the second and third plurality ofimages, and/or the first and third plurality of images. The processedinformation corresponding to the analysis of the first, second, and/orthird plurality of images may be combined. In some embodiments,processing unit 110 may perform a combination of monocular and stereoimage analyses. For example, processing unit 110 may perform monocularimage analysis (e.g., via execution of monocular image analysis module402) on the first plurality of images and stereo image analysis (e.g.,via execution of stereo image analysis module 404) on the second andthird plurality of images. The configuration of image capture devices122, 124, and 126—including their respective locations and fields ofview 202, 204, and 206—may influence the types of analyses conducted onthe first, second, and third plurality of images. The disclosedembodiments are not limited to a particular configuration of imagecapture devices 122, 124, and 126, or the types of analyses conducted onthe first, second, and third plurality of images.

In some embodiments, processing unit 110 may perform testing on system100 based on the images acquired and analyzed at steps 710 and 720. Suchtesting may provide an indicator of the overall performance of system100 for certain configurations of image capture devices 122, 124, and126. For example, processing unit 110 may determine the proportion of“false hits” (e.g., cases where system 100 incorrectly determined thepresence of a vehicle or pedestrian) and “misses.”

At step 730, processing unit 110 may cause one or more navigationalresponses in vehicle 200 based on information derived from two of thefirst, second, and third plurality of images. Selection of two of thefirst, second, and third plurality of images may depend on variousfactors, such as, for example, the number, types, and sizes of objectsdetected in each of the plurality of images. Processing unit 110 mayalso make the selection based on image quality and resolution, theeffective field of view reflected in the images, the number of capturedframes, the extent to which one or more objects of interest actuallyappear in the frames (e.g., the percentage of frames in which an objectappears, the proportion of the object that appears in each such frame,etc.), and the like.

In some embodiments, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images bydetermining the extent to which information derived from one imagesource is consistent with information derived from other image sources.For example, processing unit 110 may combine the processed informationderived from each of image capture devices 122, 124, and 126 (whether bymonocular analysis, stereo analysis, or any combination of the two) anddetermine visual indicators (e.g., lane markings, a detected vehicle andits location and/or path, a detected traffic light, etc.) that areconsistent across the images captured from each of image capture devices122, 124, and 126. Processing unit 110 may also exclude information thatis inconsistent across the captured images (e.g., a vehicle changinglanes, a lane model indicating a vehicle that is too close to vehicle200, etc.). Thus, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images based onthe determinations of consistent and inconsistent information.

Navigational responses may include, for example, a turn, a lane shift, achange in acceleration, and the like. Processing unit 110 may cause theone or more navigational responses based on the analysis performed atstep 720 and the techniques as described above in connection with FIG. 4. Processing unit 110 may also use data derived from execution ofvelocity and acceleration module 406 to cause the one or morenavigational responses. In some embodiments, processing unit 110 maycause the one or more navigational responses based on a relativeposition, relative velocity, and/or relative acceleration betweenvehicle 200 and an object detected within any of the first, second, andthird plurality of images. Multiple navigational responses may occursimultaneously, in sequence, or any combination thereof.

Smart Lane Offset

System 100 may provide driver assist functionality that monitors andadjusts the free space between vehicle 200 and boundaries (e.g., laneboundaries) within which vehicle 200 is traveling. A lane may refer to adesignated or intended travel path of a vehicle and may have marked(e.g., lines on a road) or unmarked boundaries (e.g., an edge of a road,a road barrier, guard rail, parked vehicles, etc.). For instance, as adefault, system 100 may maximize or increase the free space betweenvehicle 200 and the current lane boundaries. Further, under certainconditions, system 100 may offset vehicle 200 toward one side of thecurrent lane of travel without leaving the lane. For example, wherevehicle 200 is traveling around a curve, system 100 may decrease theoffset towards the inside of the curve (e.g., adjust the position ofvehicle 200 so that it is closer to the inside of the curve). As anotherexample, where obstacles—such as parked cars, pedestrians, orcyclists—are present on one side of the lane, system 100 may increasethe offset on the side of vehicle 200 where the obstacles are located(e.g., adjust the position of vehicle 200 so that it is further from theobstacles).

FIG. 8A illustrates vehicle 200 traveling on a roadway 800 in which thedisclosed systems and methods for identifying lane constraints andoperating vehicle 200 within the lane constraints may be used. As shown,vehicle 200 may have a first vehicle side 802, which may be a firstdistance 805 from a first lane constraint 810. Similarly, vehicle 200may have a second vehicle side 812 opposite from first vehicle side 802,and second vehicle side 812 may be a second distance 815 from secondlane constraint 820. In this manner, first lane constraint 810 andsecond lane constraint 820 may define a lane within which vehicle 200may travel.

Processing unit 110 may be configured to determine first lane constraint810 and second lane constraint 820 based on a plurality of imagesacquired by image capture device 122-126 that processing unit 110 mayreceive via data interface 128. According to some embodiments, firstlane constraint 810 and/or second lane constraint 820 may be identifiedby visible lane boundaries, such as dashed or solid lines marked on aroad surface. Additionally or alternatively, first lane constraint 810and/or second lane constraint 820 may include an edge of a road surfaceor a barrier. Additionally or alternatively, first lane constraint 810and/or second lane constrain 820 may include markers (e.g., Botts'dots). According to some embodiments, processing unit 110 may determinefirst lane constraint 810 and/or second lane constraint 820 byidentifying a midpoint of a road surface width. According to someembodiments, if processing unit 110 identifies only first laneconstraint 810, processing unit 110 may estimate second lane constraint820, such as based on an estimated lane width or road width. Processingunit 110 may identify lane constraints in this manner when, for example,lines designating road lanes are not painted or otherwise labeled.

Detection of first lane constraint 810 and/or second lane constraint 820may include processing unit 110 determining their 3D models in cameracoordinate system. For example, the 3D models of first lane constraint810 and/or second lane constraint 820 may be described by a third-degreepolynomial. In addition to 3D modeling of first lane constraint 810and/or second lane constraint 820, processing unit 110 may performmulti-frame estimation of host motion parameters, such as the speed, yawand pitch rates, and acceleration of vehicle 200. Optionally, processingunit 110 may detect static and moving vehicles and their position,heading, speed, and acceleration, all relative to vehicle 200.Processing unit 110 may further determine a road elevation model totransform all of the information acquired from the plurality of imagesinto 3D space.

Generally, as a default condition, vehicle 200 may travel relativelycentered within first and second lane constraints 810 and/or 820.However, in some circumstances, environmental factors may make thisundesirable or unfeasible, such as when objects or structures arepresent on one side of the road. Thus, there may be circumstances inwhich it may be desirable or advantageous for vehicle 200 to be closerto one lane constraint or the other. Processing unit 110 may also beconfigured to determine whether such circumstances, called lane offsetconditions, exist based on the plurality of images.

To determine whether lane offset conditions exist, processing unit 110may be configured to determine the presence of one or more objects 825in the vicinity of vehicle 200. For example object 825 may compriseanother vehicle, such as a car, truck or motorcycle traveling on roadway800. Processing unit 110 may determine, for each object 825, an offsetprofile. The offset profile may include a determination of whether inits current and predicted position object 825 will be within apredefined range of vehicle 200. If processing unit 110 determines thatobject 825 is or will be within a predefined range of vehicle 200,processing unit 110 may determine whether there is enough space withinfirst lane constraint 810 and second lane constraint 820, for vehicle200 to bypass object 825. If there is not enough space, processing unit110 may execute a lane change. If there is enough space, processing unit110 may determine whether there is enough time to bypass object 825before the distance between vehicle 200 and object 825 closes. If thereis enough time, processing unit 110 may initiate and or activate theoffset maneuver to bypass object 825. The offset maneuver may beexecuted by processing unit 110 so that the movement of vehicle 200 issmooth and comfortable for the driver. For example, the offset slew ratemay be approximately 0.15 to 0.75 m/sec. The offset maneuver may beexecuted so that the maximum amplitude of the offset maneuver will occurwhen the gap between vehicle 200 and object 825 closes. If there is notenough time, processing unit 110 may initiate braking (e.g., bytransmitting electronic signals to braking system 230) and then initiateoffset maneuver.

After each offset profile is set for each object 825, processing unitmay combine offset profiles such that the maximum lateral distance iskept from each object 825 that vehicle 200 is bypassing. That is, offsetmaneuvers may be modified and/or executed such that the lateral distancebetween vehicle 200 and each object 825 that vehicle 200 is bypassing ismaximized.

Additionally or alternatively, object 825 may be a parked or stationaryvehicle, a wall, or a person, such as pedestrian or cycling traffic.Processing unit 110 may determine a lane offset condition exists basedon certain characteristics of object 825. For example, processing unit110 may determine object 825 constitutes a lane offset condition if theheight of object 825 exceeds a predetermined threshold, such as 10 cmfrom a road surface. In this manner, processing unit 110 may beconfigured to filter out objects that would not present a lane offsetcondition, such as small pieces of debris.

Additionally or alternatively, processing unit 110 may be configured todetermine whether a lane offset condition exists on first vehicle side802 based on a position of a target object 830 on vehicle first side802. Target object 830 may include one or more objects that constitute alane offset condition. Additionally or alternatively, target object 830may not constitute a lane offset condition. For example, target object830 may be a street sign or other object a sufficient distance from thetravel lane or occurring with sufficient infrequency to not constitute alane offset condition. Processing unit 110 may be configured to concludea lane offset condition does not exist if such a target object 830 isdetected.

Processing unit 110 may be configured to cause vehicle 200 to travelsuch that first distance 805 and second distance 810 are substantiallydifferent to avoid or stay farther away from object 825. For example, ifa lane offset condition exists on first vehicle side 802, processingunit 110 may be configured to cause vehicle 200 to travel within firstlane constraint 810 and second lane constraint 820 such that firstdistance 805 is greater than second distance 815. This is illustrated,for example, in FIG. 8A. Alternatively, if a lane offset conditionexists on second vehicle side 812, processing unit 110 may create orexecute an offset profile that is configured to cause vehicle 200 totravel within first lane constraint 810 and second lane constraint 820such that first distance 805 is less than second distance 815. This isillustrated, for example, in FIG. 8B.

As another example, a parked vehicle may be located on one side ofvehicle 200 and a pedestrian may be located on the other side of vehicle200. In such a situation, system 100 may determine that is desirable toreduce the distance between vehicle 200 and the parked vehicle, in orderto provide more space between vehicle 200 and the pedestrian.Accordingly, processing unit 110 may cause vehicle 200 to travel suchthat the distance between the parked vehicle is reduced.

The identity of first vehicle side 802 and second vehicle side 812 neednot be static, such that at different points in time, processing unit110 may identify first vehicle side 802 as the driver's side or as thepassenger's side of vehicle 200. For example, the identity of firstvehicle side 802 may change depending on the presence of one or moreobjects 825 on that side of vehicle 200, such that if one lane offsetcondition exists based on object 825, it exists on first vehicle side802. In such circumstances, processing unit 110 may create or execute anoffset profile that is configured to cause vehicle 200 to travel withinfirst lane constraint 810 and second lane constraint 820 such that firstdistance 805, which is greater than second distance 815, is providedtoward object 825.

According to some embodiments, when a lane offset condition exists onfirst vehicle side 802, processing unit 110 may create an offset profileconfigured to cause vehicle 200 to travel within first lane constraint810 and second lane constraint 820 such that first distance 805 is atleast 1.25 times greater than second distance 815. Additionally oralternatively, when a lane offset condition exists on first vehicle side802, processing unit 110 may create an offset profile that is configuredto cause vehicle 200 to travel within first lane constraint 810 andsecond lane constraint 820 such that first distance 805 is at least 2times greater than second distance 815.

In some embodiments, as shown in FIG. 8C, processing unit 110 maydetermine a lane offset condition exists on both first vehicle side 802and on second vehicle side 812. Processing unit 110 may then create anoffset profile for this offset condition that is configured to causevehicle 200 to travel within first lane constraint 810 and second laneconstraint 820 such that first distance 805 is substantially the same assecond distance 815.

Processing unit 110 may be configured so that the offset profiles formoving objects 825 as well as stationary objects 825 are prioritized.For example, processing unit 110 may prioritize offset profiles to keepa predefined distance between vehicle 200 and moving objects 825 and/orstatic objects 825. Processing unit 200 may prioritize these offsetconditions over those offset conditions based on certain lanecharacteristics, such as a curve. The last priority lane offsetcondition may include mimicking a lead vehicle.

Processing unit 110 may factor offset profiles or offset conditions intoa desired look-ahead point, which may be located at the middle of thelane defined by first lane constraint 810 and second lane constraint820. Processing unit 110 may compute and implement a yaw rate commandbased on the look-ahead point and the offset profiles. The yaw ratecommand may be used to compute a torque command based on the yaw ratecommand and the yaw rate of vehicle 200 measured by using a gyroscope.Processing unit 110 may transmit the torque command to a steering system(e.g., steering system 240) of vehicle 200.

According to some embodiments, input from a driver of vehicle 200, suchas via user interface 170, may override processing unit 110. Forexample, a user input may override the control of processing unit 110 tocause vehicle 200 to travel within first lane constraint 810 and secondlane constraint 820 such that first distance 805 is greater than seconddistance 815.

Additionally or alternatively, processing unit 110 may be configured todetermine, based on the plurality of images captured by image capturedevice 122, whether a lane offset condition is substantiallynon-existent. This may include circumstances in which one or moreobjects 825 below a certain size threshold are detected, one or moreobjects 825 detected are outside of a predetermined distance from thelane constraint, and/or when no objects 825 are detected. Under suchcircumstances, processing unit 110 may be configured to cause vehicle200 to travel within first lane constraint 810 and second laneconstraint 820 such that first distance 805 is equal to or substantiallyequal to second distance 815.

A lane offset condition may include a curved road, as illustrated inFIG. 8D. Processing unit 110 may be configured to cause vehicle 200 totravel within first lane constraint 810 and second lane constraint 820such that first distance 805 is toward the outside of the curved roadrelative to vehicle 200 and second distance 815 is toward an inside ofthe curved road relative to vehicle 200. For example, first distance 805may be less than second distance 815.

FIG. 9 is an exemplary block diagram of memory 140 and/or 150, which maystore instructions for performing one or more operations consistent withdisclosed embodiments. As illustrated in FIG. 9 , memory 140 may storeone or more modules for performing the lane offset condition detectionand responses described herein. For example, memory 140 may store a laneconstraint module 910 and a lane offset module 920. Applicationprocessor 180 and/or image processor 190 may execute the instructionsstored in any of modules 910 and 920 included in memory 140. One ofskill in the art will understand that references in the followingdiscussions to processing unit 110 may refer to application processor180 and image processor 190 individually or collectively. Accordingly,steps of any of the following processes may be performed by one or moreprocessing devices.

Lane constraint module 910 may store instructions which, when executedby processing unit 110, may detect and define first lane constraint 810and second lane constraint 820. For example, lane offset module 910 mayprocess the plurality of images received from at least one image capturedevice 122-124 to detect first lane constraint 810 and second laneconstraint 820. As discussed above, this may include identifying paintedlane lines and/or measuring a midpoint of a road surface.

Lane offset module 920 may store instructions which, when executed byprocessing unit 110, may detect the presence of a lane offset conditionand/or identify whether lane offset condition is present on firstvehicle side 802 and/or second vehicle side 812. For example, laneoffset module 920 may process the plurality of images to detect thepresence of object(s) 825 on first vehicle side 802 and/or secondvehicle side 812. Lane offset module 920 may process the plurality ofimages to detect a curve in the road on which vehicle 200 is traveling.Additionally or alternatively, lane offset module 920 may receiveinformation from another module or other system indicative of thepresence of object(s) 825 and/or a curved lane. Lane offset module 920may execute control to change the position of vehicle 200 to first laneconstraint 810 and/or second lane constraint 820. For example, if laneoffset module 920 determines a lane offset condition only on firstvehicle side 802, lane offset module 920 may move vehicle 200 closer tosecond lane constraint 820. On the other hand, if lane offset module 920determines a lane offset condition only on first vehicle side 812, laneoffset module 920 may move vehicle 200 closer to first lane constraint810.

FIG. 10 illustrates a process 1000 for navigating vehicle 200,consistent with disclosed embodiments. Process 1000 may identify laneconstraints that define a lane of travel for vehicle travel. Process1000 include causing vehicle 200 to travel closer to one or the otherlane constraint, such as first lane constraint 810 or second laneconstraint 820, in response to detecting a lane offset condition. Inthis manner, process 1000 may be used to drive vehicle 200 within laneconstraints as well as account for environmental factors that may makeit more desirable to shift vehicle 200 within first lane constraint 810and second lane constraint 820.

At step 1010, process 1000 may include acquiring, using at least oneimage capture device 122, 124, and/or 126, a plurality of images of anarea in the vicinity of vehicle 200. For example, processing unit 110may receive the plurality of images may through data interface 128. Forexample, processing unit 110 may be configured to determine the presenceof one or more objects 825 on the side of the road. For example, object825 may be a parked or stationary vehicle, a wall, or a person, such aspedestrian or cycling traffic. Processing unit 110 may determine if alane offset condition exists based on certain characteristics of object825. For example, processing unit 110 may determine that object 825constitutes a lane offset condition if the height of object 825 exceedsa predetermined threshold, such as 10 cm from a road surface. In thismanner, processing unit 110 may be configured to filter out objects thatwould not present a lane offset condition, such as small pieces ofdebris.

Additionally or alternatively, processing unit 110 may be configured todetermine whether a lane offset condition exists on first vehicle side802 based on a position of a target object 830 on vehicle first side802. Target object 830 may be one or more objects that constitute a laneoffset condition. Additionally or alternatively, target object 830 maynot constitute a lane offset condition. For example, target object 830may be a road barrier or a street sign. Processing unit 110 may beconfigured to conclude a lane offset condition does not exist if such atarget object 830 is detected.

At step 1020, process 1000 may determine from the plurality of images afirst lane constraint on first vehicle side 802 and a second laneconstraint on second vehicle side 812. For example, processing unit 110may be configured to determine first lane constraint 810 and second laneconstraint 820 based on the plurality of images received via datainterface 128.

At step 1030, process 1000 may determine whether a lane offset conditionexists on first vehicle side 802. If a lane offset condition does existon first vehicle side 802, at step 1040, process 1000 may includecausing vehicle 200 to travel within first lane constraint 810 andsecond lane constraint 820 such that second distance 815 is less thanfirst distance 805. Processing unit 110 may be configured to causevehicle 200 to travel such that first distance 805 and second distance810 are substantially different to avoid or stay farther away fromobject 825. For example, if a lane offset condition exists on firstvehicle side 802, processing unit 110 may be configured to cause vehicle200 to travel within first lane constraint 810 and second laneconstraint 820 such that first distance 805 is greater than seconddistance 815. This is illustrated, for example, in FIG. 8A.Alternatively, if a lane offset condition exists on second vehicle side812, processing unit 110 may be configured to cause vehicle 200 totravel within first lane constraint 810 and second lane constraint 820such that first distance 805 is less than second distance 815. This isillustrated, for example, in FIG. 8B. For example, to navigate vehicle200 according to process 1000, processing unit 110 may transmitelectronic signals to one or more of throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200.

If a lane offset condition does not exist on first vehicle side 802, instep 1050, method may include determining whether lane offset conditionexists on second vehicle side 812. If a lane offset condition exists onsecond vehicle side 812, the method may include causing vehicle 200 totravel between first lane constraint 810 and second lane constraint 820such that first distance 805 is less than second distance 815.

Navigating a Vehicle to a Default Lane

System 100 may provide driver assist functionality that monitors thelocation of vehicle 200 in a current lane of travel and moves vehicle200 into a predetermined default travel lane if the current lane oftravel is not the predetermined default travel lane. The predetermineddefault travel lane may be defined by a user input and/or as a result ofprocessing images received from at least one image capture device122-126.

System 100 may also provide notification of an impending lane change ofvehicle 200 to the predetermined default travel lane to the driver andpassengers of vehicle 200 and/or other drivers. For example, system 100may activate a turn signal positioned at or near one of bumper regions210. Additionally or alternatively, system 100 may sound an audiblenotification, such as through speakers 360. Vehicle 200 may use system100 to select a particular lane as predetermined default travel lanewhen vehicle is traveling. When the current lane within which vehicle200 is traveling—e.g., the current lane of travel—differs from thepredetermined default travel lane, system 100 may cause vehicle 200 tomake a lane change, e.g., into the predetermined default travel lane.

FIG. 11 illustrates vehicle 200 traveling in a current lane of travel1110 of roadway 800. According to some embodiments, vehicle 200 may havea predetermined default travel lane 1120. For example, predetermineddefault travel lane 1120 may be the right-most lane of roadway 800.Navigation system 100 may be configured to determine whether vehicle 200is traveling in predetermined default travel lane 1120 and, if not,cause vehicle 200 to change to predetermined default travel lane 1120.For example, processing unit 110 may be configured to compare currentlane of travel 1110 with predetermined default travel lane 1120 andcause vehicle 200 to return to predetermined default travel lane 1120.

According to some embodiments, processing unit 110 may detect an object1130 in predetermined default travel lane 1120 based on a plurality ofimages acquired by one or more of image capture devices 122-126.Processing unit 110 may detect the position and speed of vehicle 200relative to object 1130 by analyzing the images using techniquesdescribed in connection with FIGS. 5A-5D, above. According to someembodiments, processing unit 110 may determine whether vehicle 200should pass object 1130. This may be desirable if, for example, thespeed and/or acceleration of vehicle 200 relative to object 1130 exceedsa certain threshold.

FIG. 12A is an exemplary block diagram of memory 140, which may storeinstructions for performing one or more operations consistent withdisclosed embodiments. As illustrated in FIG. 12A, memory 140 may storeone or more modules for performing the default travel lane and currenttravel lane detection and responses described herein. For example,memory 140 may store a current lane detection module 1210, a defaultlane detection module 1220, and an action module 1230.

Current lane detection module 1210 may store instructions which, whenexecuted by processing unit 110, may detect and define current lane oftravel 1110. For example, processing unit 110 may execute current lanedetection module 1212 to process the plurality of images received fromat least one image capture device 122-124 and detect current lane oftravel 1110. This may include detecting the relative position of vehicle200 with respect to the edge of roadway 800, a midpoint of roadway 800,and/or lane markers, such as painted lane lines. Processing unit 110 maydetect the position of vehicle 200 relative to roadway 800 by analyzingthe images using techniques described in connection with FIGS. 5A-5D,above.

Default lane detection module 1220 may store instructions which, whenexecuted by processing unit 110, may identify predetermined defaulttravel lane 1120 and compare default lane detection module 1220 tocurrent lane of travel 1110 identified by current lane detection module1210. Default lane detection module 1220 may identify predetermineddefault travel lane 1120 based upon a user input and/or the plurality ofimages received from at least one image capture device 122-124. Forexample, a user may select a default lane through voice commandsreceived by speakers 360 or by making a selection on a menu displayed ontouch screen 320.

In some embodiments, predetermined default travel lane 1120 may be theright-most travel lane, or it may be the lane that has the moredesirable features. For example, default lane detection module 1220 mayinclude instructions to select the lane with less traffic, the lane inwhich vehicles are traveling closest to speed of vehicle 200, and/or alane closest to an approaching exit or street to which vehicle 200 willturn. In some embodiments, predetermined default travel lane 1120 may bedetermined based on data received from position sensor 130, such as aglobal positioning system.

Action module 1230 may store instructions which, when executed byprocessing unit 110, may cause a response to the identification ofcurrent lane of travel 1110 and predetermined default travel lane 1120.For example, processing unit 110 may execute action module 1230 to causevehicle 200 to change lanes if current lane of travel 1110 is not thesame as predetermined default travel lane 1120. Additionally oralternatively, action module 1230 may initiate a notification of thelane change, such as activating turn signal 210 and or causing anaudible announcement.

Action module 1230 may include instructions that cause processing unit110 to determine whether vehicle 200 should move out of predetermineddefault travel lane 1120. For example, action module 1230 may determinethat object 1130 in predetermined default travel lane 1120 should bebypassed by vehicle 200.

Action module 1230 may further include instructions that causeprocessing unit 110 to determine whether it is safe to make a lanechange. For example, prior to executing a lane change, processing unit110 may evaluate determine one or more characteristics of roadway 800based on, for example, analysis of the plurality of images conductedusing techniques described in connection with FIGS. 5A-5D, above. If theanalysis indicates conditions are not safe to change lanes, processingunit 110 may determine that vehicle 200 should stay in the current lane.

Action module 1230 may include additional instructions for manipulatingvehicle 200 to maneuver out of and return to predetermined defaulttravel lane 1120, such as for bypassing object 1130. When executingthese instructions, processing unit 110 may estimate a location ofpredetermined default travel lane 1120 using estimated motion of vehicle200. This may be advantageous in circumstances in which predetermineddefault travel lane 1120 does not remain detectable by image capturedevices 122-124.

Processing unit 110 may be configured to execute an offset profile thatoffsets vehicle 200 from predetermined default travel lane 1120 by afull lane width, which may be estimated based on the distance betweendetected lane markers. The offset maneuver defined by offset profile maybe executed by processing unit 110 so that the movement of vehicle 200is smooth and comfortable for the driver. For example, the offset slewrate may be approximately 0.15 to 0.75 m/sec. During the offsetmaneuver, processing unit 110 may execute instructions from actionmodule 1230 that switch to following the center of current lane oftravel 1110. These instructions may include gradually transitioning fromfollowing the predicted lane with offset to following current lane oftravel 1110. A gradual transition may be accomplished by definingdesired look-ahead point so that it is not allowed to change in timemore than the recommended slew rate, which may be around 0.25 m/sec.

The instructions for transitioning vehicle 200 from predetermineddefault travel lane 1120 may be used to transition vehicle 200 back intopredetermined default travel lane. For example, processing unit 110 maybe configured to execute an offset profile that offsets vehicle 200 fromcurrent lane of travel 1110 by a full lane width, which may be estimatedbased on the distance between detected lane markers. The offset maneuverdefined by offset profile may be executed by processing unit 110 so thatthe movement of vehicle 200 is smooth and comfortable for the driver.For example, the offset slew rate may be approximately 0.15 to 0.75m/sec. During the offset maneuver, processing unit 110 may executeinstructions from action module 1230 that switch to following the centerof predetermined default travel lane 1120. These instructions mayinclude gradually transitioning from following the predicted lane withoffset to following predetermined default travel lane 1120. A gradualtransition may be accomplished by defining desired look-ahead point sothat it is not allowed to change in time more than the recommended slewrate, which may be around 0.25 m/sec.

At one or more of the steps for transitioning vehicle 200 between twolanes, processing unit 110 may compute and implement a yaw rate commandbased on the look-ahead point and the offset profiles. The yaw ratecommand may be used to compute a torque command based on the yaw ratecommand and the yaw rate of vehicle 200 measured using a gyroscope.Processing unit 110 may transmit a torque command to a steering system(e.g., steering system 240) of vehicle 200.

FIG. 12B illustrates a process 1250 for navigating vehicle 200 to adefault lane consistent with disclosed embodiments. According to someembodiments, process 1250 may be implemented by one or more componentsof navigation system 100, such as at least one processing unit 110.Process 1250 may identify current lane of travel 1110 of vehicle 200 andcompare it with predetermined default travel lane 1120 to determinewhether vehicle 200 is traveling within predetermined default travellane 1120. If vehicle 200 is not traveling in predetermined defaulttravel lane 1120, processing unit 110 may cause vehicle 200 to changelanes so that it navigates to default travel lane 1120.

At step 1260, one or more of image capture devices 122-124 may acquire aplurality of images of an area in a vicinity of vehicle 200. Processingunit 110 may receive the plurality of images via data interface 128.

At step 1270, processing unit 110 may execute current lane detectionmodule 1210 detect and define current lane of travel 1110. For example,processing unit 110 may process the plurality of images to, for example,detect the relative position of vehicle 200 with respect to the edge ofroadway 100, a midpoint of roadway 100, and/or painted lane markers.Processing unit 110 may detect the position of vehicle 200 relative toroadway 800 by analyzing the images using techniques described inconnection with FIGS. 5A-5D, above.

At step 1280, processing unit 110 may execute default lane detectionmodule 1220 to identify predetermined default travel lane 1120 andcompare default lane detection module 1220 to current lane of travel1110 identified by current lane detection module 1210. Default lanedetection module 1220 may identify predetermined default travel lane1120 based upon a user input and/or the plurality of images receivedfrom at least one image capture device 122-124.

For example, as discussed above, processing unit 110 may determinepredetermined default travel lane 1120 based on any number of factors.Predetermined default travel lane 1120 may be the right-most lane amongthe plurality of travel lanes 1110 and 1120. Additionally oralternatively, predetermined default travel lane 1120 may be determinedbased on an input received from a user via user interface 170, byprocessing the plurality of images, and/or based on data received fromposition sensor 130, such as a global positioning system. Predetermineddefault travel lane 1120 may be determined once, once per vehicle trip,or on regular intervals. For example, predetermined default travel lane1120 may be determined dynamically in response to conditions at a givenlocation.

If current lane 1110 is not the same as predetermined default travellane 1120, processing unit 110 may execute action module 1230 to causevehicle 100 to navigate to default travel lane 1120. For example,processing unit 110 may transmit electronic signals to one or more ofthrottling system 220, braking system 230, and/or steering system 240 ofvehicle 200 to trigger the desired response. For example, processingunit 110 may cause steering system 240 to turn the steering wheel ofvehicle 200 to achieve a rotation of a predetermined angle.

Further, prior to executing a lane change, processing unit 110 maydetermine whether it is safe (e.g., there are no vehicles or objects inthe way) to change lanes. As discussed above, processing unit 110 mayevaluate one or more characteristics of roadway 800 based on, forexample, analysis of the plurality of images conducted using techniquesdescribed in connection with FIGS. 5A-5D, above. If conditions are notsafe to change lanes, processing unit 110 may determine that vehicle 200should stay in the current lane.

In some embodiments, execution of action module 1230 may initiate anotification of the lane change. For example, processing unit 110 maycause a signal to activate a turn signal and/or cause an audibleannouncement via speakers 360. Accordingly, such notification may occurprior to vehicle 200 changing lanes. In other embodiments, execution ofaction module 1230 may initiate a notification to a driver of vehicle200 that vehicle 200 is not traveling in a default lane without causingthe lane change to occur. Accordingly, in such an embodiment, the driverof vehicle 200 may determine whether to manually take action and steervehicle 200 to change lanes.

In some embodiments, system 100 may notify the user of vehicle 200and/or other drivers in the vicinity of vehicle 200 that vehicle 200will be changing lanes if vehicle 200 will be moved to predetermineddefault travel lane 1120. For example, processing unit 110 may activatea turn signal of vehicle 200 prior to causing vehicle 200 to changelanes. Additionally or alternatively, an audible announcement to notifyother drivers in the vicinity may be caused by processing unit 110. Theaudible announcement may be transmitted via, for example, Bluetooth, toreceivers included in other nearby vehicles. Additionally oralternatively, at least one processing unit 110 may cause an audibleindicator to notify the driver of vehicle 200 prior to causing vehicle200 to switch lanes. In this manner, driver of vehicle 200 cananticipate the control system 100 will exert on vehicle 200.

Controlling Velocity of a Turning Vehicle

System 100 may provide driver assist functionality that controls thevelocity (i.e., speed and/or direction) of vehicle 200 in differentscenarios, such as while making a turn (e.g., through a curve). Forexample, system 100 may be configured to use a combination of map,position, and/or visual data to control the velocity of vehicle 200 whenturning. In particular, system 100 may control the velocity of vehicle200 as it approaches a curve, while navigating a curve, and/or as itexits a curve. System 100 may consider different factors to control thevelocity depending on the position of vehicle 200 relative to the curveat a given time. For example, system 100 may initially cause vehicle 200to slow down in response to learning that a curve is located a certaindistance ahead of vehicle 200. While vehicle 200 turns through thecurve, however, system 100 may cause vehicle 200 to accelerate and/orcharge direction based on an analysis of the characteristics of thecurve.

FIGS. 13A and 13B are diagrammatic representations of a vehicle (e.g.,vehicle 200) approaching, and navigating, a curve with one or morecharacteristics on road 1300, consistent with the disclosed embodiments.As illustrated in FIG. 13A, visual indicators, such as road sign 1302located near road 1300, may inform vehicle 200 that a curve lies ahead.In response to the warning provided by road sign 1302, a driver ofvehicle 200 and/or a system 100 providing driver assist functionality ofvehicle 200 may adjust the velocity of vehicle 200 (e.g., by slowingdown and/or steering) to safely navigate the curve.

As vehicle 200 approaches the curve, system 100 may detect the lanewidth 1306 and the curve radius 1308 by using image capture devices 122and/or 124 (which may be cameras) of vehicle 200. Based on the curveradius 1308, the lane width 1306, and other characteristics associatedwith the curve detected using image capture devices 122 and/or 124,system 100 may adjust the velocity of vehicle 200. Specifically, system100 may adjust the velocity as vehicle 200 approaches the curve and/orwhile vehicle 200 navigates the curve. System 100 may also adjust thevelocity as vehicle 200 exits the curve, in response to, for example,traffic light 1304 appearing after the curve. Further detail regardingadjustments to the velocity of vehicle 200 and detection of the lanewidth 1306, the radius of curvature or curve radius 1308, andcharacteristics associated with the curve are described in connectionwith FIGS. 14 and 15 , below.

In some embodiments, system 100 may recognize a curve to be navigatedbased on map data and/or vehicle position information (e.g., GPS data).System 100 may determine an initial target velocity for the vehiclebased on one or more characteristics of the curve as reflected in themap data. System 100 may adjust a velocity of the vehicle to the initialtarget velocity and determine, based on one or more images acquired byone or more of image capture devices 122-126, one or more observedcharacteristics of the curve. System 100 may determine an updated targetvelocity based on the one or more observed characteristics of the curveand adjust the velocity of the vehicle to the updated target velocity.

FIG. 14 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments. As indicated in FIG. 14 , memory 140 may store acurve recognition module 1402, a characteristic observation module 1404,and a velocity module 1406. The disclosed embodiments are not limited toany particular configuration of memory 140. Further, processing unit 110may execute the instructions stored in any of modules 1402-1406 includedin memory 140.

In one embodiment, curve recognition module 1402 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, detects a curve by, for example, analyzing one ormore sets of images and/or using map and/or position data (e.g., datastored in map database 160). The image sets may be acquired by one ormore of image capture devices 122, 124, and 126. Based on the analysis,system 100 (e.g., via processing unit 110) may cause a change invelocity of vehicle 200. For example, processing unit 110 may causevehicle 200 to reduce its speed by a predetermined amount and/or adjustits direction by a particular angle if processing unit 110 detects thepresence of a curve located within a minimum threshold distance ahead ofvehicle 200.

In one embodiment, characteristic observation module 1404 may storesoftware instructions (such as computer vision software) which, whenexecuted by processing unit 110, observes characteristics associatedwith a curve detected by system 100 (e.g., via execution of curverecognition module 1402). Processing unit 110 may observecharacteristics associated with a detected curve by analyzing one ormore sets of images acquired by one or more of image capture devices122, 124, and 126 and/or using map and/or position data (e.g., datastored in map database 160). The analysis may yield informationassociated with the curve, such as curve radius 1308, lane width 1306, adegree of curvature, a rate of change in curvature, a degree of banking,a length or arc length of the curve, and the like. These characteristicsmay include estimated values (e.g., curve characteristics calculatedusing preexisting data and/or mathematical models based on such data)and actual values (e.g., curve characteristics calculated by analyzingcaptured images of the curve). Based on the analysis, processing unit110 may cause a change in velocity of vehicle 200. For example,processing unit 110 may cause vehicle 200 to reduce its speed and/oradjust its direction by a particular angle in view of a high rate ofchange in curvature associated with a curve located 25 meters ahead ofvehicle 200.

In one embodiment, velocity module 1406 may store software instructionsconfigured to analyze data from one or more computing andelectromechanical devices configured to determine a target velocity andcause a change in velocity of vehicle 200 to the target velocity. Forexample, processing unit 110 may execute velocity module 1406 tocalculate a target velocity for vehicle 200 based on data derived fromexecution of curve recognition module 1402 and characteristicobservation module 1404. Such data may include, for example, an initialtarget position and initial velocity, an updated target position andupdated velocity, a final target position and final velocity, and thelike. In addition, processing unit 110 may calculate a velocity forvehicle 200 based on input from other systems of vehicle 200, such as athrottling system 220, braking system 230, and/or steering system 240 ofvehicle 200. Based on the calculated target velocity, processing unit110 may transmit electronic signals (e.g., via a controller area networkbus (“CAN bus”)) to throttling system 220, braking system 230, and/orsteering system 240 of vehicle 200 to trigger a change in the speedand/or direction of vehicle 200 by, for example, physically depressingthe brake, easing up off the accelerator of vehicle 200, or steeringvehicle 200 in a particular direction (e.g., at a particular angle). Anincrease in speed may be associated with a corresponding acceleration(e.g., between 0.2 m/sec² and 3.0 m/sec²). Conversely, a decrease inspeed may be associated with a corresponding deceleration. In addition,the acceleration or deceleration may be based on the road type (e.g.,highway, city street, country road, etc.), the presence of any speedconstraints nearby (e.g., sharp curves, traffic lights), as well as aroad width or lane width 1306. For purposes of this disclosure,deceleration may refer to an acceleration having a negative value.

FIG. 15 is a flow chart showing an exemplary process 1500 forcontrolling the velocity of a vehicle based on a detected curve andobserved characteristics of the curve and/or map data regarding thecurve consistent with disclosed embodiments. At step 1510, processingunit 110 may receive a plurality of images via data interface 128between processing unit 110 and image acquisition unit 120. Forinstance, a camera included in image acquisition unit 120 (such as imagecapture device 122) may capture a plurality of images and transmit themover a digital connection (e.g., USB, wireless, Bluetooth, etc.) toprocessing unit 110. In some embodiments, processing unit 110 mayreceive more than one plurality of images via a plurality of datainterfaces. For example, processing unit 110 may receive a plurality ofimages from each of image capture devices 122, 124, 126, each of whichmay have an associated data interface for communicating data toprocessing unit 110. The disclosed embodiments are not limited to anyparticular data interface configurations or protocols.

At step 1520, processing unit 110 may execute curve recognition module1502 to detect a curve located ahead of vehicle 200 based on, forexample, map and/or position data (e.g., data stored in map database160). Such data may indicate the location of a curve that is present ona given road. The data may indicate the absolute position of aparticular curve (e.g., via GPS coordinates) or the relative position ofthe curve (e.g., by describing the curve relative to vehicle 200, roadsign 1302, traffic light 1304, and/or other descriptive landmarks). Insome embodiments, processing unit 110 may execute curve recognitionmodule 1502 to detect the curve by analyzing the plurality of images.The analysis may be conducted using techniques described in connectionwith FIGS. 5A-5D and 6 , above.

At step 1530, processing unit 110 may execute velocity module 1406 todetermine an initial target velocity for vehicle 200 based on, forexample, data derived from execution of curve recognition module 1402.The initial target velocity may reflect a speed and a direction forvehicle 200 to safely enter an upcoming curve and may be determinedusing techniques described in connection with FIG. 14 , above. Forexample, velocity module 1406 may determine the initial target velocitybased on known or estimated characteristics associated with the upcomingcurve, such as the curve radius 1308 and the degree of banking. Velocitymodule 1406 may calculate the initial target velocity in accordance withmathematical models and/or formulae and based on various constraints. Insome embodiments, for example, velocity module 1406 may determine theinitial target velocity based on the presence of traffic lights (such astraffic light 1304) before, during, or after a curve, weather conditions(e.g., light rain, heavy rain, windy, clear, etc.), road conditions(e.g., paved road, unpaved road, potholes, etc.), posted speed limits,the presence of vehicles or pedestrians nearby, etc. The initial targetvelocity may also be based on a lateral acceleration limit associatedwith vehicle 200. In some embodiments, the lateral acceleration limitmay be in the range of 0.2 m/sec² and 3.0 m/sec² and may be adjusteddepending on the curve characteristics, the road width, or lane width1306. Additionally, the initial target velocity may be based on a focalrange associated with one or more of cameras, such as image capturedevices 122 and 124.

For example, velocity module 1406 may determine the initial targetvelocity based on the radius of curvature of the road ahead of vehicle200. Velocity module 1406 may determine the radius of curvature atvarious distances from vehicle 200 and store the information in a vectorR (1:n). In one embodiment, velocity module 1406 may determine theradius of curvature at: (1) the current position of vehicle 200, (2) adistance D_(end) from vehicle 200, where D_(end) represents the distanceahead of vehicle 200 at which knowledge of the road characteristicsallows system 100 to control navigational responses in vehicle 200 suchthat the driver may feel comfortable, and (3) at incremental distancesfrom vehicle 200, such as every 1 meter, etc. The value of distanceD_(end) may depend on the current speed of vehicle 200, V_(ego). Forexample, the greater the value of V_(ego) (e.g., the faster vehicle 200is traveling), the greater the value of D_(end) (e.g., the longer thedistance may be taken into account to provide system 100 with sufficienttime to provide a navigational response, such as comfortably reducingthe speed of vehicle 200 based on an approaching sharp turn or curve).In one embodiment, D_(end) may be calculated according to the followingequation: D_(end)=(V_(ego) ²/2)/(0.7/40*V_(ego)+0.5). Thus, the vector D(1:n)=[1:1:n], where n=floor(D_(end)).

In some embodiments, velocity module 1406 may determine the initialtarget velocity based on a lateral acceleration constraint, a_(max),e.g., the maximum allowed lateral acceleration. a_(max) may fall in therange of 1.5 to 3.5 meters/sec². Based on a_(max) and the radius ofcurvature vector R (discussed above), velocity module 1406 may limit thespeed of vehicle 200 (e.g., at different points or distances from thecurrent position of vehicle 200) to a speed v_(lim), calculated as:v_(lim) (1:n)=sqrt (a_(max)*R (1:n)). In addition, a decelerationconstraint that follows from the determination of v_(lim) may becalculated as: dec_(lim)=min {((v_(lim) (1:n)²−V_(ego) ²)/2), max{d_(min) D (1:n)−2*V_(ego)}}, where d_(min) is equal to 40 meters. Thevalue of d_(min) may represent a point at which deceleration feelscomfortable for the driver of vehicle 200.

In some embodiments, velocity module 1406 may determine the initialtarget velocity based on a curvature slew rate constraint, slew_(max),which may be based on the curvature rate of change. slew_(max) may fallin the range of 0.5 to 1.5 meters/sec³. Velocity module 1406 maydetermine v_(lim) according the following equation: v_(lim)(1:n)=(slew_(max)/abs{diff {1/R (1:n)}})^(1/3). Further, velocity module1406 may determine dec_(lim) according to the equation: dec_(lim)=min{((v_(lim) (1:n)²−2)/2), max {d_(min), D (1:n)−2*V_(ego)}}, whered_(min) is equal to 40 meters.

In some embodiments, velocity module 1406 may determine a comfortconstraint such that the driver of vehicle 200 is comfortable whilevehicle 200 is undergoing various navigational responses, such as makinga turn or navigating a curve. Velocity module 1406 may determine v_(lim)according to the following equation: v_(lim) (1:n)=4/500*R (1:n)+22.5.Further, velocity module 1406 may determine dec_(lim) according to theequation: dec_(lim)=min {((v_(lim) (1:n)²−V_(ego) ²)/2), max {d_(min), D(1:n)−2*V_(ego)}}, where d_(min) is equal to 40 meters.

Velocity module 1406 may merge the three constraints determinedabove—the lateral acceleration constraint, the curvature slew rateconstraint, and comfort constraint—by using the smallest value ofdec_(lim) calculated in association with each constraint. After mergingthe constraints, velocity module 1406 may arrive at the initial targetvelocity by determining how much to adjust the speed of vehicle 200,denoted v_(com), according to the following equation:v_(com)=v_(com(prev))+dec_(lim)*Δt, where v_(com(prev)) represents theprior speed of vehicle 200 (e.g., a previously determined speed, thespeed of vehicle 200 prior to any adjustment by velocity module 1406,etc.), and Δt represents the time between the determination of v_(com)and v_(com(prev)).

At step 1540, processing unit 110 may execute velocity module 1406 toadjust a velocity of vehicle 200 to the initial target velocitydetermined at step 1530. Based on the current velocity of vehicle 200and the initial target velocity determined at step 1530, processing unit110 may transmit electronic signals to throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200 to change thevelocity of vehicle 200 to the initial target velocity. Further, one ormore actuators may control throttling system 220, braking system 230,and/or steering system 240. For example, processing unit 110 maytransmit electronic signals that cause system 100 to physically depressthe brake by a predetermined amount or ease partially off theaccelerator of vehicle 200. Further, processing unit 110 may transmitelectronic signals that cause system 100 to steer vehicle 200 in aparticular direction. In some embodiments, the magnitude of acceleration(or de-acceleration) may be based on the difference between the currentvelocity of vehicle 200 and the initial target velocity. Where thedifference is large (e.g., the current velocity is 10 km/hr greater thanthe initial target velocity), for example, system 100 may change thecurrent velocity by braking in a manner that maximizes deceleration andresults in vehicle 200 achieving the initial target velocity in theshortest amount of time safely possible. Alternatively, system 100 mayapply the brakes in a manner that minimizes deceleration and results invehicle 200 achieving the initial target velocity gradually. Moregenerally, system 100 may change the current velocity to the initialtarget velocity according to any particular braking profile (e.g., abraking profile calling for a high level of braking for the first twoseconds, and a low level of braking for the subsequent three seconds).The disclosed embodiments are not limited to any particular brakingprofile or manner of braking.

At step 1550, processing unit 110 may execute characteristic observationmodule 1404 to determine one or more characteristics of a curve locatedahead of vehicle 200 based on, for example, analysis of the plurality ofimages conducted using techniques described in connection with FIGS.5A-5D, above. The characteristics may include, for example, estimatedand/or actual values associated with curve radius 1308, a road width orlane width 1306, a degree of curvature, a rate of change in curvature, adegree of banking, and the like. The characteristics may also include,for example, information conveyed by one or more road signs (such asroad sign 1302). Furthermore, the characteristics may include the lengthor arc length of the detected curve, which may be a function of thespeed of vehicle 200 as it approaches the curve. Processing unit 110 mayexecute characteristic observation module 1404 to calculate valuesassociated with these characteristics (e.g., curve radius 1308, thedegree of curvature, the rate of change in curvature, the arc length ofthe curve, etc.) based on mathematical models and/or formulae. In someembodiments, processing unit 110 may execute characteristic observationmodule 1404 to construct a mathematical model expressing the curvatureof the curve as a function of distance (e.g., distance traveled). Such amodel may be subject to constraints such as a maximum lateralacceleration constraint (e.g., in the range of 1.5 to 3.0 m/sec²) and/ora maximum lateral acceleration derivative constraint (e.g., in the rangeof 0.8 to 1.2 m/sec²). These constraints may yield a maximum speed forvehicle 200 as a function of distance traveled.

At step 1560, processing unit 110 may execute velocity module 1406 tocalculate an updated target velocity based on the characteristicsdetermined at step 1550. The updated target velocity may reflect anupdated speed and/or direction for vehicle 200 to safely perform anycombination of: (i) entering an upcoming curve, (ii) navigating througha curve, or (iii) exiting a curve. The updated target velocity may bedetermined using techniques described in connection with FIG. 14 andstep 1530, above. For example, velocity module 1406 may determine theupdated target velocity based on characteristics associated with a curveand mathematical models and/or formulae.

At step 1570, processing unit 110 may execute velocity module 1406 toadjust the velocity of vehicle 200 to the updated target velocitydetermined at step 1560. This adjustment may be accomplished usingtechniques described in connection with step 1540, above. For example,based on the current velocity of vehicle 200 and the updated targetvelocity determined at step 1560, processing unit 110 may transmitelectronic signals (e.g., via a CAN bus) to throttling system 220,braking system 230, and/or steering system 240 of vehicle 200 to changethe velocity of vehicle 200 to the updated target velocity. At thispoint, the current velocity of vehicle 200 is not necessarily the sameas the initial target velocity determined at step 1530. For example,processing unit 110 may have determined the updated target velocity atstep 1560 while in the process of adjusting the velocity of vehicle 200to the initial target velocity. This may be true in the case where thedifference in the velocity of vehicle 200 and the initial targetvelocity is large (e.g., greater than 10 km/hr) and the associatedacceleration is small.

In some embodiments, system 100 may make regular updates to the targetvelocity based on continued observations of characteristics of a curve(e.g., based on image data depicting portions of the curve that vehicle200 is approaching). Additionally, after system 100 has navigatedvehicle 200 through a curve, system 100 may cause vehicle 200 toaccelerate to a new target velocity suitable for traveling a segment ofthe road that does not have curves, as determined by analyzing map data,positional data, and/or image data acquired by system 100.

Mimicking a Leading Vehicle

System 100 may provide driver assist functionality that causes vehicle200 to mimic (or decline to mimic) a leading vehicle in differentscenarios, such as when the leading vehicle switches lanes, accelerates,or makes a turn. For example, system 100 may be configured to detect theleading vehicle by analyzing a plurality of images and determine one ormore actions taken by the leading vehicle. In some scenarios, such aswhen the leading vehicle is turning at an intersection, system 100 maybe configured to cause vehicle 200 to decline to mimic the turn. Inother scenarios, such as when the leading vehicle is turning at anintersection without changing lanes within the intersection, system 100may be configured to cause vehicle 200 to mimic the turn. System 100 mayalso be configured to cause vehicle 200 to mimic one or more actions ofthe leading vehicle based on a navigation history of the leadingvehicle.

FIGS. 16A and 16B are diagrammatic representations of a primary vehicle200 a mimicking one or more actions of a leading vehicle 200 b on road1600 a, consistent with the disclosed embodiments. As illustrated inFIGS. 16A and 16B, primary vehicle 200 a may be trailing leading vehicle200 b while traveling in the same lane of road 1600 a. The lane may havea left edge 1620, a right edge 1630, and a midpoint 1610, with w₁ and w₂indicating the left and right halves of the lane, respectively. Primaryvehicle 200 a and leading vehicle 200 b may be positioned in the lanesuch that neither vehicle is centered on the midpoint 1610. As shown inFIG. 16A, for example, primary vehicle 200 a may be centered to theright of midpoint 1610 such that the distance c₁ between primary vehicle200 a and left edge 1620 is greater than the distance c₂ between primaryvehicle 200 a and right edge 1630. In contrast, leading vehicle 200 bmay be centered to the left of midpoint 1610 such that the distance c₃between leading vehicle 200 b and left edge 1620 is less than thedistance c₄ between leading vehicle 200 b and right edge 1630. Differentconfigurations of the positions of primary vehicle 200 a and leadingvehicle 200 b are possible. For example, primary vehicle 200 a andleading vehicle 200 b may be positioned anywhere on road 1600 a.Further, although FIG. 16A depicts leading vehicle 200 b as having acamera system, the disclosed embodiments are not limited to aconfiguration in which leading vehicle 200 b includes a camera system.

System 100 of primary vehicle 200 a may cause primary vehicle 200 a tomimic one or more actions of leading vehicle 200 b. As shown in FIG.16B, for example, primary vehicle 200 a may mimic a lane shift performedby leading vehicle 200 b. After mimicking the lane shift, distance c₁may be equal to distance c₃, and distance c₂ may be equal to distancec₄. In some embodiments, primary vehicle 200 a may mimic a lane shiftperformed by leading vehicle 200 b in a way such that distances c₁ andc₃, and c₂ and c₄, are unequal. For example, in the case where primaryvehicle 200 a mimics a leftward lane shift of primary vehicle 200 b andis positioned (after the lane shift) to the left of primary vehicle 200a, c₁ will be less than c₃ and c₂ will be greater than c₄.Alternatively, where primary vehicle 200 a mimics a leftward lane shiftof primary vehicle 200 b and is positioned to the right of primaryvehicle 200 a, c₁ will be greater than c₃ and c₂ will be less than c₄.

FIGS. 16C and 16D are diagrammatic representations of a primary vehicle200 a mimicking one or more actions of a leading vehicle 200 b on road1600 b, consistent with the disclosed embodiments. As illustrated inFIG. 16C, primary vehicle 200 a may be traveling behind leading vehicle200 b, which may make a left turn at an intersection. System 100 ofprimary vehicle 200 a may cause primary vehicle 200 a to mimic the leftturn of leading vehicle 200 b, as shown in FIG. 16D. System 100 maycause primary vehicle 200 a to mimic the left turn based on variousfactors, such as whether leading vehicle 200 b changes lanes whileturning within the intersection, the navigation history associated withleading vehicle 200 b, the relative difference in speed between primaryvehicle 200 a and leading vehicle 200 b, and the like. Additionally,primary vehicle 200 a may mimic leading vehicle 200 b under variousconditions, such as based on whether leading vehicle 200 b is travelingalong the same route or the same portion of a route that primary vehicle200 a is traveling along. Further detail regarding scenarios in whichsystem 100 may cause primary vehicle 200 a to mimic one or more actionsof leading vehicle 200 b are described in connection with FIGS. 17-22 ,below.

In some embodiments, system 100 may be configured to determine whetherto mimic a particular movement of a leading vehicle. For example, system100 may determine that certain movements of the leading vehicle do notneed to be mimicked because the movements do not affect the course ofthe leading vehicle. For example, system 100 may determine that smallchanges in movements, such as moving within a particular lane, do notneed to be mimicked. In other embodiments, system 100 may implement asmoothing operation to filter out small movements of a leading vehicle.The smoothing operation may cause the primary vehicle to implement moresignificant and/or important movements (e.g., changing lanes) of aleading vehicle while filtering out smaller movements not affecting theoverall course of the leading vehicle.

FIG. 17 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments. As shown in FIG. 17 , memory 140 may store a laneconstraint module 1702, an action detection module 1704, and an actionresponse module 1706. The disclosed embodiments are not limited to anyparticular configuration of memory 140. Further, processing unit 110 mayexecute the instructions stored in any of modules 1702-1706 included inmemory 140.

In one embodiment, lane constraint module 1702 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, determine one or more constraints associated with alane that primary vehicle 200 a is traveling in. The lane constraintsmay include a midpoint 1610, a left edge 1620, and a right edge 1630 ofthe lane. The lane constraints may also include one or more lines orother symbols marked on the surface of road 1600 a or 1600 b. Processingunit 110 may execute lane constraint module 1702 to determine theconstraints by, for example, analyzing one or more sets of images and/orusing map and/or position data (e.g., data stored in map database 160).The image sets may be acquired by one or more of image capture devices122, 124, and 126. Based on the analysis, system 100 (e.g., viaprocessing unit 110) may cause primary vehicle 200 a to travel within alane defined by left edge 1620 and right edge 1630. In some embodiments,system 100 may cause primary vehicle 200 a to move away from left edge1620 or right edge 1630 if one of distance c₁ or c₂ is less than apredetermined threshold.

In one embodiment, action detection module 1704 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, detects one or more actions taken by leadingvehicle 200 b traveling in front of primary vehicle 200 a. Leadingvehicle 200 b may be traveling in the same or different lane as primaryvehicle 200 a. Processing unit 110 may execute action detection module1704 to detect one or more actions taken by leading vehicle 200 b by,for example, analyzing one or more sets of images acquired by one ormore of image capture devices 122, 124, and 126. Based on the analysis,processing unit 110 may determine, for example, that leading vehicle 200b has shifted lanes, made a turn, accelerated, decelerated, applied itsbrakes, and/or the like. As another example, leading vehicle 200 b mayperform a maneuver such that a first offset distance on a side ofleading vehicle 200 b adjacent to a first lane constraint is differentfrom a second offset distance on a side of leading vehicle 200 badjacent to a second lane constraint. For example, processing unit 110may perform the analysis based on the techniques described in connectionwith FIGS. 5A-5D, above.

In one embodiment, action response module 1706 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, determines one or more actions for primary vehicle200 a to take based on one or more actions taken by leading vehicle 200b detected by system 100 (e.g., via execution of action detection module1704). For example, processing unit 110 may execute action responsemodule 1706 to determine whether to mimic one or more actions of leadingvehicle 200 b. This determination may be based on the nature of leadingvehicle 200 b's actions (e.g., turn, lane shift, etc.), informationassociated with primary vehicle 200 a (e.g., speed, distance c₁ and c₂from the lane edges, etc.), road and environmental conditions (e.g.,potholes, heavy rain or wind, etc.), and the like. In the case whereprocessing unit 110 determines to mimic one or more actions of leadingvehicle 200 b, processing unit 110 may accomplish this by, for example,transmitting electronic signals (e.g., via a CAN bus) to throttlingsystem 220, braking system 230, and/or steering system 240 of primaryvehicle 200 a to trigger a turn, lane shift, change in speed, and/orchange in direction of primary vehicle 200 a. Processing unit 110 mayuse the techniques described in connection with FIGS. 4-7 , above, tocause primary vehicle 200 a to mimic the one or more actions. To mimicthe one or more actions, system 100 may provide control signals to oneor more of throttling system 220, braking system 230, and steeringsystem 240 to navigate vehicle 200 (e.g., by causing an acceleration, aturn, a lane shift, etc.). Further, one or more actuators may controlthrottling system 220, braking system 230, and/or steering system 240.

FIG. 18 is a flow chart showing an exemplary process for causing primaryvehicle 200 a to mimic one or more actions of leading vehicle 200 bconsistent with disclosed embodiments. At step 1810, processing unit 110may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120. For instance, acamera included in image acquisition unit 120 (such as image capturedevice 122) may capture a plurality of images and transmit them over adigital connection (e.g., USB, wireless, Bluetooth, etc.) to processingunit 110. In some embodiments, processing unit 110 may receive more thanone plurality of images via a plurality of data interfaces. For example,processing unit 110 may receive a plurality of images from each of imagecapture devices 122, 124, 126, each of which may have an associated datainterface for communicating data to processing unit 110. The disclosedembodiments are not limited to any particular data interfaceconfigurations or protocols.

At steps 1820 and 1830, processing unit 110 may execute lane constraintmodule 1702 to determine constraints for each side of primary vehicle200 a associated with a lane that primary vehicle 200 a is traveling in.Processing unit 110 may determine the constraints by analyzing theimages received at step 1810. Thus, at step 1820, processing unit 110may determine a first lane constraint on a first side of primary vehicle200 a (e.g., the distance between the left side of primary vehicle 200 aand left edge 1620 of the lane, c₁) and, at step 1830, a second laneconstraint on a second side opposite the first side (e.g., the distancebetween the right side of primary vehicle 200 a and right edge 1620 ofthe lane, c₂). In some embodiments, processing unit 110 may determinethe constraints by using map and/or position data (e.g., data stored inmap database 160) indicating the position of primary vehicle 200 arelative to midpoint 1610, left edge 1620, and right edge 1630 of thelane. Processing unit 110 may determine the constraints based on both ananalysis of the images received at step 1810 and map and/or positiondata; doing so may increase the confidence level associated with theconstraints.

At step 1840, processing unit 110 may execute lane constraint module1702 to cause primary vehicle 200 a to travel within the laneconstraints determined at steps 1820 and 1830. For example, the leftside of primary vehicle 200 a may be positioned to the left of left edge1620, or the right side of primary vehicle 200 a may be positioned tothe right of right edge 1630. In such a scenario, processing unit 110may transmit electronic signals (e.g., via a CAN bus) to throttlingsystem 220, braking system 230, and/or steering system 240 of primaryvehicle 200 a to cause primary vehicle 200 a to adjust its position andtravel within the lane constraints.

At step 1850, processing unit 110 may execute action detection module1704 to locate a leading vehicle 200 b within the images received atstep 1810 by analyzing the images. Processing unit 110 may detect thepresence of vehicles in the images using the techniques described inconnection with FIGS. 5A and 5B, above. For each detected vehicle,processing unit 110 may construct a multi-frame model of the detectedvehicle's position, velocity (e.g., speed and direction), andacceleration relative to primary vehicle 200 a. Processing unit 110 mayuse the techniques described in connection with FIGS. 5A and 5B, above,to construct such a model. In addition, processing unit 110 maydetermine a lane of travel associated with each detected vehicle, basedon, for example, lane or other road markings and lane constraintsdetermined at steps 1820 and 1830. Processing unit 110 may determineleading vehicle 200 b to be the detected vehicle that is closest to, andis traveling in the same lane as, primary vehicle 200 a. At step 1860,processing unit 110 may execute action detection module 1704 todetermine one or more actions taken by the leading vehicle 200 bdetected within the images at step 1850. For example, processing unit110 may determine that leading vehicle 200 b has shifted lanes, made aturn, accelerated, decelerated, and/or the like. In addition, processingunit 110 may analyze the images to determine characteristics associatedwith leading vehicle 200 b, such as its speed and position (e.g.,relative to primary vehicle 200 a). These determinations may be based onthe techniques described in connection with FIGS. 5A and 5B, above. Insome embodiments, processing unit 110 may analyze the images todetermine characteristics associated with an action performed by leadingvehicle 200 b. For example, where leading vehicle 200 b makes a turn,processing unit 110 may determine position information for leadingvehicle 200 b right before it begins the turn, position information forleading vehicle 200 b as it completes the turn, a turning radius, and aspeed profile describing changes in speed (e.g., over time) for leadingvehicle 200 b as it makes the turn.

At step 1870, processing unit 110 may execute action response module1706 to cause primary vehicle 200 a to mimic the one or more actions ofleading vehicle 200 b determined at step 1850. For example, processingunit 110 may transmit electronic signals (e.g., via a CAN bus) tothrottling system 220, braking system 230, and/or steering system 240 ofprimary vehicle 200 a to trigger a turn, lane shift, change in speed,and/or change in direction of primary vehicle 200 a. In addition,processing unit 110 may construct a set of path points to guide changesin position and speed for primary vehicle 200 a. The path points may bebased on lane marks (e.g., midpoint 1610, left edge 1620, right edge1630, etc.) represented by a polynomial (e.g., of the third-degree) andtarget coordinates associated with a target vehicle, such as leadingvehicle 200 b. Processing unit 110 may adjust the coefficients of thepolynomial so that the path points pass through the target coordinates.Processing unit 110 may also offset the path points (e.g., relative toleft edge 1620 and right edge 1630) based on a predetermined offsetamount. As processing unit 110 adjusts the polynomial coefficients andoffsets the path points, processing unit 110 may also calculate localcurvature information for each of the resulting segments formed betweenthe path points.

In some embodiments, processing unit 110 (e.g., via execution of actionresponse module 1706) may cause primary vehicle 200 a to mimic leadingvehicle 200 b within certain position, speed, and/or accelerationconstraints associated with the two vehicles. For example, theconstraints may require a predetermined minimum distance (e.g., a safetydistance) between primary vehicle 200 a and leading vehicle 200 b or apredetermined maximum difference in speed and/or acceleration betweenthe vehicles. Thus, if mimicking an action performed by leading vehicle200 b would violate one or more constraints (e.g., mimicking anacceleration would result in primary vehicle 200 a being too close toleading vehicle 200 b), processing unit 110 may cause primary vehicle200 a to decline to mimic the one or more actions of leading vehicle 200b. In addition, even while mimicking an action performed by leadingvehicle 200 b (e.g., a lane shift), processing unit 110 may causeprimary vehicle 200 a to follow leading vehicle 200 b's speed profileand maintain a predetermined distance or a safety distance from leadingvehicle 200 b. Processing unit 110 may accomplish this by causingprimary vehicle 200 a to: (i) continuously close the distance gap fromleading vehicle 200 b when the gap is greater than the predeterminedminimum distance, (ii) continuously close the gap between the speeds ofprimary vehicle 200 a and leading vehicle 200 b, and (iii) match theacceleration of leading vehicle 200 b to the extent it is consistentwith predetermined constraints. To control the extent to which of theseactions should be performed, processing unit 110 may associate eachaction with weights in real-time based on the current and predicteddistance to leading vehicle 200 b. Thus, for example, as the distance toleading vehicle 200 b decreases more weight may be put on deceleratingrather than matching the acceleration of leading vehicle 200 b.

As another example, in the case where primary vehicle 200 a isapproaching leading vehicle 200 b at a speed greater than the speed ofleading vehicle 200 b, processing unit 110 may cause primary vehicle 200a to gradually slow down such that, when the distance between the twovehicles equals a predetermined safety distance such as a minimumfollowing distance, the speed of the two vehicles is the same. Inanother example, where a leading vehicle 200 b abruptly appears in frontof primary vehicle 200 a, causing the distance between the two vehiclesto be less than the minimum following distance, processing unit 110 maycause primary vehicle 200 a to gradually slow down to avoid approachingleading vehicle 200 b any closer while at the same increasing thedistance between the vehicles until it reaches the minimum followingdistance.

FIG. 19 is a flow chart showing an exemplary process 1900 for causingprimary vehicle 200 a to decline to mimic a turn of leading vehicle 200b, consistent with disclosed embodiments. At step 1910, processing unit110 may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120, as described inconnection with step 1810 of FIG. 18 , above. At step 1920, processingunit 110 may execute action detection module 1704 to locate a leadingvehicle 200 b within the images received at step 1910 by analyzing theimages, as described in connection with step 1850 of FIG. 18 , above. Atstep 1925, processing unit 110 may execute action detection module 1704to determine whether leading vehicle 200 b is turning at anintersection, using the techniques described in connection with step1860 of FIG. 18 , above. If leading vehicle 200 b is determined to beturning at the intersection (step 1925, yes), processing unit 110 maycause primary vehicle 200 a to decline to mimic the turn of leadingvehicle 200 b at step 1930. Otherwise (step 1925, no), process 1900concludes.

FIG. 20 is a flow chart showing an exemplary process 2000 for causingprimary vehicle 200 a to mimic or decline a turn of leading vehicle 200b, consistent with disclosed embodiments. At step 2010, processing unit110 may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120, as described inconnection with step 1810 of FIG. 18 , above. At step 2020, processingunit 110 may execute action detection module 1704 to locate a leadingvehicle 200 b within the images received at step 2010 by analyzing theimages, as described in connection with step 1850 of FIG. 18 , above. Atstep 2025, processing unit 110 may execute action detection module 1704to determine whether leading vehicle 200 b is turning at anintersection, using the techniques described in connection with step1860 of FIG. 18 , above. If leading vehicle 200 b is determined to beturning at the intersection (step 2025, yes), at step 2035, processingunit 110 may execute action detection module 1704 to determine whetherleading vehicle 200 b is changing lanes within the intersection, usingthe techniques described in connection with step 1860 of FIG. 18 ,above. Otherwise (step 2025, no), process 2000 concludes. If leadingvehicle 200 b is determined to be changing lanes within the intersection(step 2035, yes), processing unit 110 may cause primary vehicle 200 a todecline to mimic the turn of leading vehicle 200 b at step 2050.Otherwise (step 2035, no), processing unit 110 may cause primary vehicle200 a to mimic the turn of leading vehicle 200 b at step 2040.

FIG. 21 is a flow chart showing another exemplary process 2100 forcausing primary vehicle 200 a to mimic one or more actions of a firstleading vehicle, consistent with disclosed embodiments. At step 2110,processing unit 110 may receive a plurality of images via data interface128 between processing unit 110 and image acquisition unit 120, asdescribed in connection with step 1810 of FIG. 18 , above. At steps 2120and 2130, processing unit 110 may execute action detection module 1704to locate a first and second leading vehicle within the images receivedat step 2110 by analyzing the images, as described in connection withstep 1850 of FIG. 18 , above. At step 2135, processing unit 110 mayexecute action detection module 1704 to determine whether the firstleading vehicle path has a lower turning radius than the second leadingvehicle path, using the techniques described in connection with step1860 of FIG. 18 , above. If the first leading vehicle path is determinedto have a lower turning radius than the second leading vehicle path(step 2135, yes), processing unit 110 may cause primary vehicle 200 a tomimic the turn of the first leading vehicle at step 2140. Otherwise(step 2135, no), process 2100 concludes.

FIG. 22 is a flow chart showing an exemplary process 2200 for causingprimary vehicle 200 a to mimic one or more actions of leading vehicle200 b based on a navigation history, consistent with disclosedembodiments. At step 2210, processing unit 110 may receive a pluralityof images via data interface 128 between processing unit 110 and imageacquisition unit 120, as described in connection with step 1810 of FIG.18 , above. At step 2220, processing unit 110 may execute actiondetection module 1704 to locate a leading vehicle 200 b within theimages received at step 2210 by analyzing the images, as described inconnection with step 1850 of FIG. 18 , above.

At step 2230, processing unit 110 may determine position information forleading vehicle 200 b by analyzing the images received at step 2210.Position information for leading vehicle 200 b may indicate the positionof leading vehicle 200 b relative to primary vehicle 200 a. Processingunit 110 may analyze the images to derive position information forleading vehicle 200 b and one or more actions taken by leading vehicle200 b by using the techniques described in connection with FIGS. 5A-5D,above.

At step 2240, processing unit 110 may compare the position informationfor leading vehicle 200 b determined at step 2230 to predetermined mapdata (e.g., data stored in map database 160). For example, by comparingthe position information to predetermined map data, processing unit 110may derive position coordinates for leading vehicle 200 b. In addition,processing unit 110 may use the predetermined map data as an indicatorof a confidence level associated with the position informationdetermined at step 2230.

At step 2250, processing unit 110 may create a navigation historyassociated with leading vehicle 200 b by, for example, tracking theactions taken by leading vehicle 200 b. Processing unit 110 mayassociate the actions taken by leading vehicle 200 b (e.g., determinedfrom analysis of the images at step 2230) with predetermined map datafrom step 2240. Thus, the navigation history may indicate that leadingvehicle 200 b performed certain actions (e.g., a turn, a lane shift,etc.) at certain position coordinates.

At step 2260, processing unit 110 may execute action response module1706 to determine whether to cause primary vehicle 200 a to mimic one ormore actions of leading vehicle 200 b based on the navigation historycreated at step 2250 and using the techniques described in connectionwith step 1870 of FIG. 18 , above. For example, processing unit 110(e.g., via action response module 1706) may cause primary vehicle 200 ato mimic a turn of leading vehicle 200 b, even where no turn is detectedfrom analysis of the images (e.g., at step 2230), when the navigationhistory indicates that leading vehicle 200 b has successfully navigatedat least one prior turn. More generally, processing unit 110 may causeprimary vehicle 200 a to mimic an action taken by leading vehicle 200 bin cases where the navigation history indicates the leading vehicle 200b had successfully performed such an action.

Navigating a Vehicle to Pass Another Vehicle

System 100 may provide driver assist functionality that monitors thevicinity around vehicle 200 and aborts lane changes in the event thatthe lane change is deemed to be at risk of causing a collision, such aswith another vehicle (e.g., a target vehicle). For example, system 100may enable vehicle 200 to complete a pass of a target vehicle if thetarget vehicle is determined to be in a different lane than vehicle 200.If, before completion of a pass, system 100 determines that the targetvehicle is entering the lane in which vehicle 200 is traveling, system100 may cause vehicle 200 to abort the pass. For example, aborting thepass may include causing vehicle 200 to brake or causing vehicle 200 tostop accelerating.

As illustrated in FIG. 23 , vehicle 200 may travel on roadway 2300.Vehicle 200 may be equipped with system 100 and may implement any of thedisclosed embodiments for identifying lane constraints and operatingvehicle 200 within the lane constraints. System 100 may also facilitatelane changes made by vehicle 200. For example, system 100 may determinewhether circumstances allow vehicle 200 to safely change lanes and, ifthe circumstances change during the lane change, cause vehicle 200 toabort lane change.

As shown in FIG. 23 and as previously discussed with respect to FIG. 8 ,vehicle 200 may include first vehicle side 802, which may be firstdistance 805 from a first lane constraint 2302. Similarly, vehicle 200may include second vehicle side 812 opposite from first vehicle side802, and second vehicle side 812 may be second distance 815 from asecond lane constraint 2304. In this manner, first lane constraint 810and second lane constraint 2304 may define a lane 2330 within whichvehicle 200 may travel.

System 100 may determine first lane constraint 2302 and second laneconstraint 2304 based on a plurality of images acquired by one or moreof image capture devices 122-126. According to some embodiments, firstlane constraint 2302 and/or second lane constraint 2304 may beidentified by visible lane boundaries, including lines marked on a roadsurface. Additionally or alternatively, first lane constraint 2302and/or second lane constraint 2304 may include an edge of a roadsurface. According to some embodiments, system 100 may determine firstlane constraint 2302 and/or second lane constraint 2304 by identifying amidpoint of a road surface width 2320 (or any other suitable roadfeature that vehicle 200 may use as a navigation reference). System 100may identify lane constraints in this manner when, for example, linesdesignating road lanes are not painted or otherwise labeled.

When passing a target vehicle 2310, the lane change of vehicle 200 maybe complicated by a change in the trajectory of target vehicle 2330.Thus, system 100 may identify target vehicle 2310 based on an analysisof a plurality of images acquired by one or more of image capturedevices 122-126. System 100 may also identify second lane 2340 in whichtarget vehicle 2310 is traveling. Further, system 100 may determinewhether second lane 2340 is different than lane 2330 in which vehicle200 is traveling. If second lane 2340 is different than lane 2330,system 100 may be configured to enable vehicle 200 to pass targetvehicle 2310.

System 100 may also monitor target vehicle 2310 based on the pluralityof images. For example, system 100 may monitor a position of targetvehicle 2310, such as monitoring its position relative to vehicle 200,relative to first lane constraint 2302 and/or second lane constraint2304, an absolute position, or relative to another reference point.Additionally or alternatively, monitoring a position of target vehicle2310 may include estimating a speed of target vehicle 2310.

If system 100 determines that target vehicle 2310 is entering lane 2330in which vehicle 200 is traveling, system 100 may cause vehicle 200 toabort the pass of target vehicle 2310. For example, system 100 may causevehicle 200 to stop accelerating or brake. Additionally oralternatively, system 100 may issue an audible announcement, such as tonotify driver of vehicle 200 that the lane change is being aborted.

FIG. 24A is an exemplary block diagram of memory 140, which may storeinstructions for performing one or more operations consistent withdisclosed embodiments. As illustrated in FIG. 24A, memory 140 may storeone or more modules for identification of the target vehicle andresponses described herein. For example, memory 140 may store a laneconstraint module 2400, a target vehicle acquisition module 2410, and anaction module 2415.

In one embodiment, lane constraint module 2400 may store instructionswhich, when executed by processing unit 110, may detect and define firstlane constraint 2302 and second lane constraint 2304. For example,processing unit 110 may execute lane offset module 2400 to process theplurality of images received from at least one image capture device122-124 and detect first lane constraint 2302 and second lane constraint2304. As discussed above, this may include identifying painted lanelines or markers and/or measuring a midpoint of a road surface.Processing unit 110 may perform the analysis based on the techniquesdescribed in connection with FIGS. 5A-5D, above.

In some embodiments, target acquisition module 2410 may storeinstructions which, when executed by processing unit 110, may detect thepresence of target vehicle 2310 and monitor the position of targetvehicle 2310. For example, target acquisition module 2410 may processthe plurality of images to detect and monitor target vehicle 2310.Additionally or alternatively, when executing target acquisition module2410, processing unit 110 may receive information from another module orother system indicative of the presence and/or location of targetvehicle 2310. Processing unit 110 may perform the analysis based on thetechniques described in connection with FIGS. 5A-5D, above.

Processing unit 110 may be configured to determine the presence of oneor more objects such as target vehicle 2310 in the vicinity of vehicle200. For example target vehicle 2310 may comprise another vehicle, suchas a car, truck or motorcycle traveling near vehicle 200. Processingunit 110 may determine, for each target vehicle 2310, an offset profile.The offset profile may include a determination of whether in its currentand predicted position, target vehicle 2310 will be within a predefinedrange of vehicle 200. If processing unit 110 determines that targetvehicle 2310 is or will be within a predefined range of vehicle 200,processing unit may determine whether there is enough space within firstlane constraint 810 and second lane constraint 820, for vehicle 200 tobypass target vehicle 2310. If there is not enough space, processingunit 110 may execute a lane change. If there is enough space, processingunit 110 may determine whether there is enough time to bypass targetvehicle 2310 before the distance between vehicle 200 and target vehicle2310 closes. If there is enough time, processing unit 110 may initiateand or activate the offset maneuver to bypass target vehicle 2310. Theoffset maneuver may be executed by processing unit 110 so that themovement of vehicle 200 is smooth and comfortable for the driver. Forexample, the offset slew rate may be approximately 0.15 to 0.75 m/sec.The offset maneuver may be executed so that the maximum amplitude of theoffset maneuver will occur when the gap between vehicle 200 and object825 closes. If there is not enough time, processing unit 110 mayinitiate braking and then initiate an offset maneuver.

Further, processing unit 110 may execute action module 2415 to abort theplan of vehicle 200 to pass target vehicle 2310. If target vehicle 2310is not in lane 2330 of vehicle 200, instructions included in actionmodule 1230 may enable vehicle 200 to pass target vehicle 2330. However,if execution of target vehicle acquisition module 2410 indicates thattarget vehicle 2310 is changing lanes into lane 2330 of vehicle 200,processing unit 110 may execute instructions included in action module2415 to abort the lane change of vehicle 200. For example, processingunit 110 may transmit electronic signals to throttling system 220,braking system 230, and/or steering system 240 of vehicle 200 to abortthe lane change. Additionally or alternatively, processing unit maycause an audible notification to sound.

FIG. 24B illustrates a flowchart of an exemplary process 2420 fornavigating a vehicle among encroaching vehicles, consistent withdisclosed embodiments. Process 2420 may identify lane constraints thatdefine a lane of travel for a vehicle (e.g., vehicle 200) to travel,identify and monitor a target vehicle (e.g., vehicle 2310), and enablethe vehicle to pass the target vehicle or abort the pass if it isdetermined that the target vehicle is entering a lane in which thevehicle is traveling.

At step 2430, at least one of image capture device 122, 124, and/or 126may acquire a plurality of images of an area in the vicinity of vehicle200. For example, a camera included in image acquisition unit 120 (suchas image capture device 122) may capture a plurality of images andtransmit them over a digital connection (e.g., USB, wireless, Bluetooth,etc.) to processing unit 110. In some embodiments, processing unit 110may receive more than one plurality of images via a plurality of datainterfaces. For example, processing unit 110 may receive a plurality ofimages from each of image capture devices 122, 124, 126, each of whichmay have an associated data interface for communicating data toprocessing unit 110. The disclosed embodiments are not limited to anyparticular data interface configurations or protocols.

Next, at step 2440, processing unit 110 may determine a first laneconstraint on first vehicle side 802 and a second lane constraint onsecond vehicle side 812 using the plurality of images received via datainterface 128. For example, as part of implementing step 2440,processing unit 110 may execute instructions of lane constraint module910. Further, as part of determining the first and second laneconstraints, processing unit 110 may use one or more of the processesdiscussed above in connection with FIGS. 5A-5D.

At step 2450, processing unit 110 may acquire target vehicle 2310. Toacquire target vehicle 2310, processing unit 110 may execute targetvehicle acquisition module 2410. For example, processing unit 110 mayexecute instructions to analyze acquired images to identify second lane2340 in which target vehicle 2310 is traveling and determine whethertarget vehicle 2310 is traveling in lane 2330 in which vehicle 200 istraveling. This determination may be based on the techniques describedin connection with FIGS. 5A-5D, above.

At step 2455, processing unit 110 may determine whether second lane 2340in which target vehicle 2310 is traveling is different than lane 2330 inwhich vehicle 200 is traveling.

At step 2460, processing unit 110 may enable vehicle 200 to pass targetvehicle 2310 if second lane 2340 is different than lane 2330 in whichvehicle 200 is traveling. To enable vehicle 200 to pass target vehicle2310, processing unit may execute action module 2415.

Next, at step 2465, processing unit 110 may execute target acquisitionmodule 2410 to monitor target vehicle 2310. This may include estimatinga velocity of target vehicle 2310 and determining the location of targetvehicle 2310, such as with respect to vehicle 200.

Step 2470 may include determining whether target vehicle 2310 hasentered lane 2330. If processing unit 110 determines that target vehicle2310 has not entered lane 2330, at step 2480, processing unit 110 mayallow vehicle 200 to complete the pass.

At step 2490, if processing unit 110 determines that target vehicle 2310has entered lane 2330 (or is entering lane 2330 or is otherwise on atrajectory to bring vehicle 2310 into lane 2330), processing unit 110may execute action module 2415 to abort the pass. This may include, forexample, issuing an audible notification to driver of vehicle 200 viaspeakers 360. Additionally or alternatively, step 2490 may includeexecuting instructions included in action module 2415 to causeprocessing unit 110 to transmit electronic signals to throttling system220, braking system 230, and/or steering system 240 of vehicle 200 toabort lane change.

Navigating a Vehicle to Avoid Encroaching Traffic

System 100 may provide driver assist functionality that monitors thevicinity around vehicle 200 and responds to the presence of anothervehicle encroaching upon vehicle 200. Monitoring the vicinity aroundvehicle 200 may include monitoring a collision threshold. For example acollision threshold may include a time to collision or a minimumpredetermined distance between vehicle 200 and other traffic. System 100may further cause vehicle 200 to take evasive action to avoidencroaching traffic. For example, evasive action may include braking,changing lanes, or otherwise altering the course of vehicle 200.

As illustrated in FIG. 25 , vehicle 200 may travel on roadway 2500.Vehicle 200 may include system 100 to detect and respond to encroachingtraffic. For example, vehicle 200 may determine that another vehicle isentering into the lane in which vehicle 200 is traveling or otherwisedetermine that another vehicle is crossing a collision threshold.Vehicle 200 may react to an encroaching vehicle, such as by alteringcourse, braking, or accelerating.

As discussed with respect to FIG. 8 and as shown in FIG. 25 , vehicle200 may include first vehicle side 802, which may be first distance 805from a first lane constraint 2502. Similarly, vehicle 200 may includesecond vehicle side 812 opposite from first vehicle side 802, and secondvehicle side 812 may be second distance 815 from a second laneconstraint 2504. In this manner, first lane constraint 2502 and secondlane constraint 2504 may define a lane 2505 within which vehicle 200 maytravel.

System 100 may determine first lane constraint 2502 and second laneconstraint 2504 based on a plurality of images acquired by one or moreof image capture devices 122-126. According to some embodiments, firstlane constraint 2502 and/or second lane constraint 2504 may beidentified by visible lane boundaries, such a line marked on a roadsurface. Additionally or alternatively, first lane constraint 2502and/or second lane constraint 2504 may include an edge of a roadsurface. According to some embodiments, system 100 may determine firstlane constraint 2502 and/or second lane constraint 2502 by identifying amidpoint 2520 of a road surface width (or any other suitable roadfeature vehicle 200 may use as a navigation reference). System 100 mayidentify lane constraints in this manner when, for example, linesdesignating road lanes are not painted or otherwise labeled.

Detection of first lane constraint 2502 and/or second lane constraint2504 may include processing unit 110 determining their 3D models in acamera coordinate system. For example, the 3D models of first laneconstraint 2502 and/or second lane constraint 2504 may be described by athird-degree polynomial. In addition to 3D modeling of first laneconstraint 2502 and/or second lane constraint 2504, processing unit 110may estimate motion parameters, such as the speed, yaw and pitch rates,and acceleration of vehicle 200. Optionally, processing unit may detectstatic and moving vehicles and their position, heading, speed, andacceleration, all relative to vehicle 200. Processing unit 110 maydetermine a road elevation model to transform the information acquiredfrom the plurality of images into 3D space.

The driver of vehicle 200 cannot assume that other traffic traveling inadjacent or near-adjacent lanes will only change lanes if it is safe forthat traffic to shift into lane 2505 in which vehicle 200 is driving.Thus, it may be desirable to consider and monitor traffic traveling innearby lanes to detect lateral encroachment into lane 2505 of vehicle200, so that vehicle 200 may react to lateral encroachment to avoid acollision. System 100 may identify an encroaching vehicle 2510 based onthe plurality of images and determine that encroaching vehicle 2510 isapproaching vehicle 200. System 100 may then cause an action to accountfor the encroaching vehicle. For example, system 100 may cause vehicle200 to maintain a current velocity and travel within first laneconstraint 2502 and second lane constraint 2504 such that a firstdistance 2530, which is on first vehicle side 802 on which encroachingvehicle 2510 is approaching, is greater than a second distance 2540.

Processing unit 110 may be configured to determine the presence of oneor more encroaching vehicles 2510 in the vicinity of vehicle 200.Processing unit 110 may determine, for each encroaching vehicle 2510 anoffset profile. The offset profile may include a determination ofwhether in its current and predicted position, encroaching vehicle 2510will be within a predefined range of vehicle 200. If processing unit 110determines that encroaching vehicle 2510 is or will be within apredefined range of vehicle 200, processing unit may determine whetherthere is enough space within first lane constraint 2502 and second laneconstraint 2504, for vehicle 200 to bypass encroaching vehicle 2510. Ifthere is not enough space, processing unit 110 may execute a lanechange. If there is enough space, processing unit 110 may determinewhether there is enough time to bypass encroaching vehicle 2510 beforethe distance between vehicle 200 and encroaching vehicle 2510 closes.

For example, if there is enough time, processing unit 110 may initiateand or activate the offset maneuver to bypass encroaching vehicle 2510.The offset maneuver may be executed by processing unit 110 so that themovement of vehicle 200 is smooth and comfortable for the driver. Forexample, the offset slew rate may be approximately 0.15 to 0.75 m/sec.The offset maneuver may be executed so that the maximum amplitude of theoffset maneuver will occur when the gap between vehicle 200 andencroaching vehicle 2510 closes. If there is not enough time, processingunit 110 may initiate braking (e.g., via transmitting electronic signalsto braking system 230) and then initiate offset maneuver (e.g., bytransmitting electronic signals to throttling system 220 and/or steeringsystem 240).

System 100 may determine, based on a plurality of images acquired by oneor more of image capture devices 122-126, that encroaching vehicle 2510has crossed at least one collision threshold. For example, the at leastone collision threshold may include a minimum predetermined distance2550 between vehicle 200 and encroaching vehicle 2510. For example,encroaching vehicle 2510, as illustrated in FIG. 25 , is shown exceedingminimum predetermined distance 2550. Additionally or alternatively, theat least one collision threshold may include a time to collision ofvehicle 200 and encroaching vehicle 2510. The time-to-collisionthreshold may be determined based on the distance between vehicle 200and encroaching vehicle 2510, the velocity of vehicle 200 and/orencroaching vehicle 2510, the lateral velocity of vehicle 200 and/orencroaching vehicle 2510, and/or the acceleration of vehicle 200 and/orencroaching vehicle 2510. A collision threshold based on a time tocollision may be set to allow enough time for system 100 to effectivelyconduct evasive action to avoid collision.

If system 100 determines that at least one collision threshold has beencrossed, system 100 may cause vehicle 200 to take evasive action. Forexample, this may include causing other subsystems of vehicle 200 tooperate. Thus, system 100 may operate to cause vehicle 200 to increasespeed by communicating with throttling system 220, braking system 230,and/or steering system 240 to change the speed and/or direction ofvehicle 200.

The evasive action may include causing vehicle 200 to change lanes. Thismay be a desirable response if, for example, vehicle 200 is traveling ona multilane highway and system 100 determines that changing lanes tomove away from encroaching vehicle 2510 will avoid a collision. Evasiveaction may include vehicle 200 altering its course. Additionally oralternatively, evasive action may include altering course of vehicle200. For example, to take evasive action, processing unit 110 maytransmit electronic signals to throttling system 220, braking system230, and/or steering system 240.

System 100 may also be configured to cause vehicle 200 to take an actionafter the evasive action has been completed. For example, system 100 maycause vehicle 200 to return to midpoint 2520 of lane 2505 in whichvehicle 200 is traveling. Additionally or alternatively, system 100 maycause vehicle 200 to resume a preset speed after braking in response toencroaching vehicle 2510.

FIG. 26 is an exemplary block diagram of memory 140, which may storeinstructions for detecting and responding to traffic laterallyencroaching on a vehicle consistent with disclosed embodiments. Asillustrated in FIG. 26 , memory 140 may store a lane constraint module2610, an encroaching vehicle acquisition module 2620, and an actionmodule 2630.

In one embodiment, lane constraint module 2610 may store instructionswhich, when executed by processing unit 110, may detect and define firstlane constraint 2502 and second lane constraint 2504. For example,processing unit 110 may execute lane offset constraint module 2610 toprocess the plurality of images received from at least one image capturedevice 122-126 and detect first lane constraint 2502 and second laneconstraint 2504. As discussed above, this may include identifyingpainted lane lines and/or measuring a midpoint of a road surface.Processing unit 110 may perform the analysis based on the techniquesdescribed in connection with FIGS. 5A-5D, above.

In one embodiment, encroaching vehicle acquisition module 2620 may storeinstructions which, when executed by processing unit 110, may detect thepresence of encroaching vehicle 2510 and monitor encroaching vehicle2510. For example, encroaching vehicle acquisition module 2620 mayinclude instructions for determining a relative location of encroachingvehicle 2510 to vehicle 200. In some embodiments, this may includemonitoring the velocity of encroaching vehicle 200. According to someembodiments, encroaching vehicle acquisition module 2620 may determinewhether encroaching vehicle 2510 has exceeded a collision threshold. Acollision threshold may be, for example, a minimum predetermineddistance between vehicle 200 and encroaching vehicle 2510. Ifencroaching vehicle 2510 comes closer to vehicle 200 than the minimumpredetermined distance, processing unit 110 may determine thatencroaching vehicle 2510 is indeed encroaching and execute instructionsto avoid collision with encroaching vehicle 2510.

Additionally or alternatively, a collision threshold may include a timeto collision. The time-to-collision threshold may be determined based onthe distance between vehicle 200 and encroaching vehicle 2510, thevelocity of vehicle 200 and/or encroaching vehicle 2510, the lateralvelocity of vehicle 200 and/or encroaching vehicle 2510, and/or theacceleration of vehicle 200 and/or encroaching vehicle 2510. A collisionthreshold based on a time to collision may be set to allow enough timefor processing unit 110 and system 100 to effectively conduct evasiveaction to avoid collision.

Further, processing unit 110 may execute action module 2630 to respondto the detection of encroaching vehicle 2510 and/or the crossing of acollision threshold. For example, action module 2630 may includeinstructions to cause vehicle 200 to maintain a current velocity and totravel within first lane constraint 2502 and second lane constraint 2504such that first distance 2530 is greater than second distance 2540. Thefirst distance 2530 may be on the same side of vehicle 200 asencroaching vehicle 2510, so that vehicle 200 is farther away fromencroaching vehicle 2510 while still within first lane constraint 2502and second lane constraint 2504. If processing unit 110 determines thata collision threshold has been crossed, processing unit 110 may executeaction module 2630 to cause vehicle 200 to take evasive action, such asaltering course, braking, and/or changing lanes. Action module 2630 mayalso include instructions that cause vehicle 200 to resume course aftercompleting evasive action. For example, processing unit 110 may causevehicle 200 to resume a preset speed after braking or return to thecenter of lane 2505.

FIG. 27 illustrates a flowchart of an exemplary process 2720 detectingand responding to traffic laterally encroaching on a vehicle, consistentwith disclosed embodiments. Process 2720 may identify lane constraintsthat define a lane of travel for vehicle travel, determine whether anencroaching vehicle 2510 is approaching, and cause vehicle 200 tomaintain current velocity and to travel within first lane constraint2501 and second lane constraint 2504 such that first distance 2530, onthe side of vehicle 200 that encroaching vehicle 2510 is approaching, isgreater than second distance 2540.

At step 2730, at least one image capture device 122, 124, and/or 126 mayacquire a plurality of images of an area in the vicinity of vehicle 200.For example, a camera included in image acquisition unit 120 (such asimage capture device 122) may capture a plurality of images and transmitthem over a digital connection (e.g., USB, wireless, Bluetooth, etc.) toprocessing unit 110. In some embodiments, processing unit 110 mayreceive more than one plurality of images via a plurality of datainterfaces. For example, processing unit 110 may receive a plurality ofimages from each of image capture devices 122, 124, 126, each of whichmay have an associated data interface for communicating data toprocessing unit 110. The disclosed embodiments are not limited to anyparticular data interface configurations or protocols.

Next, at step 2740, processing unit 110 may determine a first laneconstraint on first vehicle side 802 and a second lane constraint onsecond vehicle side 812 using the plurality of images received via datainterface 128. For example, as part of implementing step 2740,processing unit 110 may execute instructions of lane constraint module2610. Further, as part of determining the first and second laneconstraints, processing unit 110 may use one or more of the processesdiscussed above in connection with FIGS. 5A-5D.

At step 2750, processing unit 110 may determine whether encroachingvehicle 2510 is approaching. To make this determination, processing unit110 may execute encroaching vehicle acquisition module 2620. Encroachingvehicle acquisition module 2620 may include instructions to processimages to detect other traffic. This may include identifying vehicleswithin a certain range of vehicle 200, such as those vehicles that areadjacent to vehicle 200. These adjacent vehicles may be encroachingvehicles 2510.

Once encroaching vehicle 2510 is identified, processing unit maydetermine more information regarding vehicle 2510. For example, at step2555, processing unit 110 may determine whether target vehicle 2330 hasexceeded a collision threshold. A collision threshold may be, forexample, a minimum predetermined distance between vehicle 200 andencroaching vehicle 2510. If encroaching vehicle 2510 comes closer tovehicle 200 than the minimum predetermined distance, processing unit 110may determine that encroaching vehicle 2510 is indeed encroaching andexecute instructions to avoid collision with encroaching vehicle 2510.

Additionally or alternatively, a collision threshold may be a time tocollision. The time-to-collision threshold may be determined based onthe distance between vehicle 200 and encroaching vehicle 2510, thevelocity of vehicle 200 and/or encroaching vehicle 2510, the lateralvelocity of vehicle 200 and/or encroaching vehicle 2510, and/or theacceleration of vehicle 200 and/or encroaching vehicle 2510. A collisionthreshold based on a time to collision may be set to allow enough timefor processing unit 110 and system 100 to effectively conduct evasiveaction to avoid collision.

At step 2760, processing unit 110 may cause vehicle 200 to maintaincurrent velocity and to travel within first lane constraint 2502 andsecond lane constraint 2504. To cause vehicle 200 to operate in thismanner, processing unit 110 may execute action module 2630. Additionallyor alternatively, step 2760 may include executing instructions on actionmodule 1230 to cause processing unit 110 to transmit electronic signalsto the accelerator 2610, brakes 2620, and/or steering system of vehicle200 to cause vehicle 200 to maintain current velocity and to travelwithin first and second lane constraints 810 and 820.

If, at step 2755, processing unit 110 determined that a collisionthreshold has been exceeded, the processing unit 110 may take evasiveaction. The evasive action may include causing vehicle 200 to changelanes. This may be a desirable response if, for example, vehicle 200 istraveling on a multilane highway and system 100 determines that changinglanes to move away from encroaching vehicle 2510 will avoid a collision.Evasive action may include vehicle 200 altering its course. Additionallyor alternatively, evasive action may include altering course of vehicle200. For example, processing unit 110 may undertake evasive action bytransmitting electronic signals to throttling system 220, braking system230, and/or steering system 240 of vehicle 200. After the evasive actionis completed, at step 2780, processing unit 110 may control vehicle 200to resume its prior navigation (e.g., to resume navigation to a desiredlocation).

Multi-Threshold Reaction Zone for Vehicle Navigation

System 100 may provide driver assist functionality that causes aresponse in vehicle 200, such as braking, accelerating, switching lanes,turning, and/or other navigational responses. For example, if system 100detects the presence of another vehicle ahead of vehicle 200 that istraveling at a rate of speed slower than vehicle 200, system 100 maycause vehicle 200 to reduce its speed by braking. System 100 may apply abraking profile that accounts for the immediacy of the risk (e.g., therisk of collision) to provide the driver with a natural drivingsensation.

System 100 may cause various different responses depending on thesituation and a determined risk of collision with another vehicle, forexample. System 100 may take no action in response to changes in speedor direction of a target vehicle ahead if the target vehicle issufficiently far ahead and the determined risk of collision is low.Where the target vehicle may be closer or when the risk of collisionrises above a predetermined level, system 100 may cause a response, suchas braking, etc. When the risk of collision is even higher (e.g., wheresystem 100 determines that a collision would be imminent without evasiveaction), system 100 may cause vehicle 100 to take an evasive action,such as maximum or near maximum braking, change of direction, etc.

FIGS. 28A and 28B are diagrammatic representations of a primary vehicle200 a on a road 2800, consistent with disclosed embodiments. In someinstances, road 2800 may be divided into two lanes, lane 2810 and lane2820. A target object, such as target vehicle 200 b, may be travelingahead of primary vehicle 200 a in the same lane (e.g., lane 2820 asshown in FIGS. 28A and 28B) or in a different lane. As shown in theexample depicted in FIG. 28A, primary vehicle 200 a and target vehicle200 b may be separated by a distance d₁. At a given time after the scenedepicted in FIG. 28A, primary vehicle 200 a and target vehicle 200 b maybe separated by a distance d₂ as shown in FIG. 28B, where d₂ is lessthan d₁. For example, primary vehicle 200 a may be traveling at a rateof speed greater than target vehicle 200 b, causing the distance betweenthe vehicles to decrease from d₁ to d₂.

As another example, primary vehicle 200 a and target vehicle 200 b maybe traveling at the same rate of speed until target vehicle 200 breduces its speed by braking, or primary vehicle 200 a increases itsspeed by accelerating, causing the distance between the vehicles todecrease from d₁ to d₂. As another example, target vehicle 200 b mayencroach from a side of primary vehicle 200 a. In some circumstances, iftarget vehicle 200 b is ahead or a sufficient distance away from primaryvehicle 200 a (e.g., five meters or more, 10 meters or more, etc.) andmoves toward primary vehicle 200 a, system 100 may not take any action.However, if target vehicle 200 b continues to move closer to primaryvehicle 200 a, then system 100 may cause primary vehicle 200 a to brake,change speed, and/or change lanes. For example, if target vehicle 200 bis within a predetermined threshold (e.g., within five meters), primaryvehicle 200 a may then take action. Different scenarios involving therelative speeds and positions of primary vehicle 200 a and targetvehicle 200 b are possible, and the disclosed embodiments are notlimited to any particular scenario.

In some embodiments, the multi-threshold reaction zone may provide anatural feeling driving experience to the user. For example, asdiscussed above, if the risk of collision is low, then system 100 maydisregard movements by target vehicle 200 b. As a result, primaryvehicle 200 a may not respond to movements or actions of target vehicle200 b when there is not a risk of a collision (e.g., primary vehicle 200a would not need to mimic every motion of target vehicle 200 b when it's100 meters ahead). However, if target vehicle 200 b is close to primaryvehicle 200 a and/or if system 100 determines that the risk of collisionis high, then a responsive maneuver may be taken such that the userfeels safer by increasing space between primary vehicle 200 a and targetvehicle 200 b. If system 100 determines that a crash may be imminent,then system 100 may take a more dramatic action without causing the userto be surprised, because the action may be necessary to avoid acollision.

Accordingly, in some embodiments, system 100 may associate the differentreaction zones with different degrees of responsive actions. Forexample, in some embodiments, system 100 may take no action when targetvehicle 200 b is within a first threshold (e.g., when target vehicle 200b is far away from primary vehicle 200 a), system 100 may take a mediumaction with target vehicle 200 b is within a second threshold (e.g.,when target vehicle 200 b is becoming closer to primary vehicle 200 a),and system 100 may take a more dramatic action when target vehicle 200is within a third threshold (e.g., when target vehicle 200 b issufficiently close to primary vehicle 200 a that there is a risk of acollision). In a first reaction zone associated with the firstthreshold, system 100 may take no action when the estimateddistance-to-target associated with target vehicle 200 b is greater thantwice a predetermined safety distance. In some embodiments, the firstreaction zone may apply where the estimated distance-to-targetassociated with target vehicle 200 b falls in the range of 1.5 to 2.5times the predetermined safety distance.

The estimated distance-to-target may be the current distance betweenprimary vehicle 200 a and target vehicle 200 b plus the expected changeof this distance over a prediction time (e.g., in the range of 0.1 to1.0 seconds, depending on factors associated with system 100 such asvelocity tracking control loops, actuators, and vehicle dynamics). Theexpected change of distance may be calculated as:(v_(target)−v_(primary))*t_(prediction)+((a_(target)−a_(primary))*t_(prediction)^(1/2)), where v_(target) and v_(primary) correspond to the speed oftarget vehicle 200 b and primary vehicle 200 a, respectively, a_(target)and a_(primary) correspond to the acceleration of target vehicle 200 band primary vehicle 200 a, respectively, and t_(prediction) correspondsto the prediction time. Choosing a prediction time approximating theoverall reaction time of system 100 may account for the fact thatdesired navigational responses for primary vehicle 200 a may not beimplemented immediately, and thereby may provide the driver and/orpassengers of primary vehicle 200 a with a smooth, natural drivingexperience.

The predetermined safety distance may be defined as: max {d_(static),d_(dynamic)}. d_(static), which may fall in the range of 2 to 10 meters,may represent a desired safety distance between primary vehicle 200 aand another vehicle (such as target vehicle 200 b) while the vehiclesare stopping and/or moving at a low rate of speed. d_(dynamic) mayrepresent a desired safety distance when the vehicles are moving at aspeed greater than a low rate of speed, and may be calculated as:t_(safety)*min (v_(target), v_(primary)), where v_(target) andv_(primary) correspond to the speed of target vehicle 200 b and primaryvehicle 200 a, respectively. t_(safety) may fall in the range of 0.5 to2.5 seconds and may be adjusted by the driver of primary vehicle 200 avia user interface 170 according to the driver's preferences. Thus, thedriver may be able to control the distance that system 100 maintainsbetween primary vehicle 200 a and other vehicles.

In a second reaction zone associated with the second threshold, system100 may place target vehicle 200 b in a safety zone and apply differentweights to different maneuvers performed by target vehicle 200 b. Forexample, as target vehicle 200 b may approach a lower boundary of thesafety zone, e.g., the estimated distance-to-target associated withtarget vehicle 200 b may approach the predetermined safety distance(discussed above in connection with the first reaction zone), system 100may assign more weight to deceleration maneuvers (e.g., mimicked more)and less weight to acceleration maneuvers (e.g., mimicked less). In someembodiments, the lower boundary may fall in the range of 0.5 to 1.5times the predetermined safety distance, depending on driverpreferences. Conversely, as target vehicle 200 b approaches an upperboundary of the safety zone, e.g., the estimated distance-to-targetassociated with target vehicle 200 b may approach twice thepredetermined safety distance, system 100 may assign less weight todeceleration maneuvers (e.g., mimicked less) and more weight toacceleration maneuvers (e.g., mimicked more). In some embodiments, thelower boundary may fall in the range of 1.5 to 2.5 times thepredetermined safety distance, depending on driver preferences.

In a third reaction zone associated with the third threshold, system 100may place target vehicle 200 b in a danger zone when the estimateddistance-to-target associated with target vehicle 200 b is less than thepredetermined safety distance. In this scenario, system 100 may causeprimary vehicle 200 a to mimic (e.g., immediately) any decelerationmaneuvers (while disregarding any acceleration maneuvers) performed bytarget vehicle 200 b. In some embodiments, the first reaction zone mayapply where the estimated distance-to-target associated with targetvehicle 200 b falls in the range of 0.5 to 1.5 times the predeterminedsafety distance, depending on driver preferences.

In some embodiments, system 100 may cause a response in primary vehicle200 a, such as braking. As indicated in FIG. 28B, for example, thedistance between the primary vehicle 200 a and target vehicle 200 b maydecrease from d₁ to d₂. Depending on the distance d₂ and, by extension,the time before collision, system 100 may cause primary vehicle 200 a toreduce its speed so as to avoid a collision with target vehicle 200 b.In addition, system 100 may apply one or more braking profiles whencausing primary vehicle 200 a to reduce its speed, also based on, forexample, the time before collision. Thus, if the time before collision(and/or the distance d₂) exceeds a first predetermined threshold, system100 may cause primary vehicle 200 a to brake in a gradual manner, changelanes, and/or change speed, whereas if the time before collision (and/orthe distance d₂) is below a second predetermined threshold, system 100may cause primary vehicle 200 a to brake, change lanes, and/or changespeed in a more rapid manner.

FIG. 29 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations, consistent with thedisclosed embodiments. As shown in FIG. 29 , memory 140 may store atarget monitoring module 2902, an intercept time module 2904, and anaction response module 2906. The disclosed embodiments are not limitedto any particular configuration of memory 140. Further, processing unit110 may execute the instructions stored in any of modules 2902-2906included in memory 140.

In one embodiment, target monitoring module 2902 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, identifies and monitors a target object (such astarget vehicle 200 b) appearing in a plurality of images representing anenvironment in the vicinity of primary vehicle 200. For purposes of thisdisclosure, the target object may also be referred to as target vehicle200 b, although the target object may be a pedestrian, road hazard(e.g., large debris), and the like. The plurality of images may beacquired by one or more of image capture devices 122, 124, and 126.Processing unit 110 may execute target monitoring module 2902 toidentify and monitor target vehicle 200 b by, for example, analyzing theplurality of images using the techniques described in connection withFIGS. 5A-5D, above. Thus, target monitoring module 2902 may trackdifferent variables, such as the motion (e.g., direction of travel andtrajectory), speed, and acceleration associated with target vehicle 200b, as well as the distance between primary vehicle 200 a and the targetvehicle 200 b. Such information may be tracked continuously or atpredetermined intervals.

In one embodiment, intercept time module 2904 may store softwareinstructions which, when executed by processing unit 110, determines anindicator of an intercept time between primary vehicle 200 a and targetvehicle 200 b. The intercept time may refer to an amount of time beforeprimary vehicle 200 a makes contact (e.g., collides) with target vehicle200 b. Processing unit 110 may execute intercept time module 2904 todetermine an indicator of the intercept time based on an analysis ofimages acquired by one or more of image capture devices 122, 124, and126. For example, intercept time module 2904 may make the determinationusing a series of time-based observations, e.g., of the position, speed,and/or acceleration of target vehicle 200 b (relative to primary vehicle200 a), such as Kalman filters or linear quadratic estimation (LQE). TheKalman filters may be based on a measurement of the scale of targetvehicle 200 b, where the scale measurement is proportional to theintercept time.

In one embodiment, action response module 2906 may store softwareinstructions (such as computer vision software) which, when executed byprocessing unit 110, causes one or more responses in primary vehicle 200a. The responses may be based on the identification and monitoring oftarget vehicle 200 b (e.g., performed via execution of target monitoringmodule 2902) and/or the indicator of an intercept time between primaryvehicle 200 a and target vehicle 200 b (e.g., determined via executionof intercept time module 2904). Processing unit 110 may cause one ormore responses in primary vehicle 200 a by, for example, transmittingelectronic signals (e.g., via a CAN bus) to throttling system 220,braking system 230, and/or steering system 240 of primary vehicle 200 ato brake, accelerate, or trigger a turn, lane shift, and/or change indirection of primary vehicle 200 a. Processing unit 110 may use extendedKalman Filters to estimate data associated with target vehicle 200 b andprimary vehicle 200 a, such as: speed, acceleration, turn rate, pitchangle, pitch rate, position and heading of target vehicle 200 b relativeto primary vehicle 200 a, the size of target vehicle 200 b, roadgeometry, and the like. For example, processing unit 110 may receivemeasurements from components on primary vehicle 200 a, such as anodometer, an inertial measurement unit (IMU), image processor 190, andthe like. Thus, processing unit 110 may calculate data associated withtarget vehicle 200 b and/or primary vehicle 200 a using state equationsderived from kinematic equations of motion, rigidity constraints, andmotion assumptions, where inputs to the equations may includemeasurements from components on primary vehicle 200 a. Processing unit110 may also use the techniques described in connection with FIGS. 4-7 ,above, to cause one or more responses in primary vehicle 200 a.

FIG. 30 is a flow chart showing an exemplary process 3000 for causing aresponse in a primary vehicle, consistent with disclosed embodiments. Atstep 3010, processing unit 110 may receive a plurality of images viadata interface 128 between processing unit 110 and image acquisitionunit 120. For instance, one or more cameras included in imageacquisition unit 120 may capture a plurality of images of an areaforward of primary vehicle 200 a (or to the sides or rear of a vehicle,for example) and transmit them over a data connection (e.g., digital,wired, USB, wireless, Bluetooth, etc.) to processing unit 110.

At step 3020, processing unit 110 may execute target monitoring module2902 to identify a target object (such as target vehicle 200 b) withinthe plurality of images. For example, target monitoring module 2902 maydetect the presence of vehicles in the images using the techniquesdescribed in connection with FIGS. 5A and 5B, above. Multiple vehicles(and/or objects) may be detected in the same lane or in a different laneas primary vehicle 200 a, and at varying distances from primary vehicle200 a, based on image data, position data (e.g., GPS locationinformation), map data, the yaw rate of primary vehicle 200 a, lane orother road markings, speed data, and/or data from sensors included inprimary vehicle 200 a. In a scenario where multiple vehicles aredetected, target monitoring module 2902 may determine target vehicle 200b to be the vehicle traveling in the same lane as primary vehicle 200 aand/or located closest to primary vehicle 200 a.

At step 3030, processing unit 110 may execute target monitoring module2902 to monitor target vehicle 200 b identified at step 3020. Forexample, target monitoring module 2902 may track information associatedwith target vehicle 200 b, such as the motion (e.g., direction of traveland trajectory), speed, and acceleration, as well as the distancebetween primary vehicle 200 a and the target vehicle 200 b. Targetmonitoring module 2902 may track such information continuously, or atpredetermined intervals, by analyzing the plurality of images using thetechniques described in connection with FIGS. 5A-5D, above. At step3040, processing unit 110 may execute intercept time module 2904 todetermine an indicator of an intercept time between primary vehicle 200a and target vehicle 200 b. Intercept time module 2904 may make thedetermination by tracking the position, speed, and/or acceleration oftarget vehicle 200 b (relative to primary vehicle 200 a) over one ormore periods of time. Intercept time module 2904 may use the techniquesdescribed in connection with FIGS. 5A and 5B, above to track suchinformation.

At step 3050, processing unit 110 may execute action response module2906 to cause one or more responses in primary vehicle 200 a. Forexample, processing unit 110 may transmit electronic signals (e.g., viaa CAN bus) to throttling system 220, braking system 230, and/or steeringsystem 240 of primary vehicle 200 a to brake, accelerate, or trigger aturn, lane shift, and/or change in direction of primary vehicle 200 a.Further, one or more actuators may control throttling system 220,braking system 230, and/or steering system 240. For example, processingunit 110 may transmit electronic signals that cause system 100 tophysically depress the brake by a predetermined amount or ease partiallyoff the accelerator of vehicle 200. Further, processing unit 110 maytransmit electronic signals that cause system 100 to steer vehicle 200in a particular direction. Such responses may be based on the monitoringof target vehicle 200 b (performed at step 3030) and/or the indicator ofan intercept time between primary vehicle 200 a and target vehicle 200 b(determined at step 3040). In some embodiments, the intercept time maybe compared with one or more predetermined thresholds, e.g., a first andsecond intercept threshold. For example, the response may include anemergency avoidance action if the indicator of intercept time indicatesthat a target object is within a particular intercept threshold (e.g.,within the second intercept threshold). The emergency avoidance actionmay include changing lanes and/or emergency braking.

The thresholds may be expressed as units of time. Thus, for example, ifthe indicator of intercept time indicates that target vehicle 200 b isoutside of the first threshold (e.g., the time before collision betweenprimary vehicle 200 a and target vehicle 200 b exceeds the firstthreshold), then action response module 2906 may cause a response inprimary vehicle 200 a based on an averaged motion profile associatedwith target vehicle 200 b. The averaged motion profile associated withtarget vehicle 200 b may represent an averaged position and/or averagedspeed of target vehicle 200 b for a particular threshold. In someembodiments, if target vehicle 200 b is a certain distance and/ortraveling a certain speed, system 100 may elect to take no action. Forexample, if target vehicle 200 b is ahead or a sufficient distance awayfrom primary vehicle 200 a (e.g., five meters, 10 meters, etc.), system100 may not take any action unless target vehicle 200 b becomes closerto primary vehicle 200.

If the indicator of intercept time indicates that target vehicle 200 bis between the first and second thresholds (e.g., the time beforecollision between primary vehicle 200 a and target vehicle 200 b fallsin between the first and second thresholds), then action response module2906 may cause a response in primary vehicle 200 a based on the actualmotion associated with target vehicle 200 b. The actual motion mayrepresent a position and/or speed of target vehicle 200 b (e.g., notaveraged or delayed) If the indicator of intercept time indicates thattarget vehicle 200 b is within the second threshold (e.g., the timebefore collision between primary vehicle 200 a and target vehicle 200 bis below the second threshold), then action response module 2906 maycause an emergency action in primary vehicle 200 a (such as a lanechange or emergency braking).

FIG. 31 is a flow chart showing an exemplary process 3100 for decliningto cause a response in a primary vehicle, consistent with disclosedembodiments. At step 3110, processing unit 110 may receive a pluralityof images via data interface 128 between processing unit 110 and imageacquisition unit 120, as described in connection with step 3010 of FIG.30 , above.

At step 3120, processing unit 110 may execute target monitoring module2902 to locate a target object (e.g., target vehicle 200 b) within theimages received at step 3010 by analyzing the images, as described inconnection with step 3020 of FIG. 30 , above. At step 3125, processingunit 110 may execute target monitoring module 2902 to determine whetherthe target object (e.g., target vehicle 200 b) is traveling in a lanedifferent from primary vehicle 200 a, using the techniques described inconnection with step 3030 of FIG. 30 , above. If target vehicle 200 b isdetermined to be traveling in a lane different from primary vehicle 200a (step 3125, yes), processing unit 110 may decline to cause a responsein primary vehicle 200 a at step 3130. Alternatively, if system 100determines that target vehicle 200 b does not constitute an imminentrisk of collision (e.g., target vehicle 200 b a nonintersectingtrajectory with primary vehicle 200 a or a potential time to collisionbeyond a threshold such that the target object does not pose an imminentrisk at the present time), processing unit 110 may decline to cause aresponse in primary vehicle 200 a at step 3130. Otherwise (step 3125,no), process 3100 may conclude and no response occurs.

Stopping an Autonomously Driven Vehicle Using a Multi-Segment BrakingProfile

System 100 may provide driver assist functionality that monitors andresponds to environmental conditions to control the braking of vehicle200. For instance, system 100 may detect, based on a plurality ofimages, an object that triggers the stopping of vehicle 200. Forexample, this object may include another vehicle in the vicinity ofvehicle 200, a traffic light, and/or a traffic sign. System 100 may usea braking profile to slow and/or stop vehicle 200. According to someembodiments, the braking profile may include multiple segments, wheredeceleration is executed at different stages. For example, the brakingprofile may cause system 100 to stop vehicle 200 at a decreasing rate ofdeceleration.

By providing a braking profile that includes multiple segments, system100 may cause vehicle 200 to operate in a manner that feels natural to adriver and/or passengers. For example, when a driver stops a vehicle ata traffic light, the brakes are typically not applied at a constant ratefrom the moment they are applied until the moment when the vehiclestops. Operating brakes in such a manner result in an abrupt stop at theend. Instead, the driver may apply the brakes gently at first, then withincreasing pressure, and then as the vehicle reaches its target stoppinglocation, the driver may decrease the level of braking until it reacheszero (or near zero) at the moment the car stops. Operating the vehiclein this manner may provide a natural stopping progression, i.e., thedriver may cause-a progressive deceleration at first so the vehicle doesnot experience an abrupt initial deceleration and then at the end, sothat vehicle does not come to an abrupt stop.

System 100 may provide autonomous vehicle navigation that may implementbraking according to an approach that feels natural to the driver and/orpassengers of vehicle 200. For example, in some embodiments, system 100may cause one or more of image capture devices 122-126 to acquire aplurality of images of an area in a vicinity of vehicle 200. Processingdevice 110 may receive the receive the plurality of images via datainterface 128 and identify, based on analysis of the plurality ofimages, a trigger (e.g., an object, a traffic light, and/or a trafficsign) for stopping vehicle 200. Based on the identified trigger, system100 may cause vehicle 200 to stop according to a braking profile. Insome embodiments, the braking profile may include a plurality ofsegments (e.g., including a first segment associated with a firstdeceleration rate, a second segment which includes a second decelerationrate less than the first deceleration rate, and a third segment in whicha level of braking is decreased as a target stopping location isapproached, as determined based on the analysis of the plurality ofimages.

FIG. 32A illustrates vehicle 200 traveling on a roadway 3210 thatincludes an object 3220. System 100 may identify a trigger (e.g., object3220) for stopping vehicle 200 in respect to detection of the trigger.For example, system 100 may first slow vehicle 200 after the trigger isdetected and bring vehicle 200 to a stop before reaching object 3220.

Processing unit 110 may be configured to determine a trigger forstopping vehicle 200 based on a plurality of images acquired by one ormore of image capture devices 122-126. Processing unit 110 may receivethe images from image capture devices 122-126 via data interface 128.According to some embodiments, the trigger may include object 3220. Forexample, object 3220 may be another vehicle on roadway 2810. Accordingto some embodiments, processing unit 110 may identify a trigger ifobject 3220 is within a certain vicinity of vehicle 200. For example,identification of the trigger may depend on a relative distance betweenobject 3220 and vehicle 200. Further, this identification may optionallyconsider the relative velocity and/or relative acceleration betweenobject 3220 and vehicle 200.

Additionally or alternatively, the trigger may include a traffic sign,such as traffic sign 3240 shown in FIG. 32A. For example, traffic sign3240 may be a stop sign and/or a speed limit sign. According to someembodiments, processing unit 110 may process the plurality of images todetermine the content of traffic sign 3240 as part of determiningwhether a trigger exists. For example, processing unit 110 may determinethat a trigger is present based on a speed limit specified by trafficsign 3240 if vehicle 200 is exceeding the speed limit. As anotherexample, processing unit 110 may determine a trigger exists based on atraffic light 2830 if the yellow and/or red light is on.

Based on the plurality of images, processing unit 110 may determine atarget stopping location 3250 at which vehicle 200 will stop once abraking profile is executed. Reaching target stopping location 3250 maybe accompanied by a release in braking pressure of vehicle 200. Targetstopping location 3250 may be based on the location of traffic light3230 and/or traffic sign 3240 such that vehicle 200 complies withtraffic light 3230 and/or traffic sign 3240. Alternatively, targetstopping location 3250 may be based on object 3220 such that vehicle 200stops before reaching object 3220. Accordingly, target stopping location3250 may be based on the location of object 3220 to avoid a collisionbetween vehicle 200 and object 3220. In some embodiments, targetstopping location 3250 may be updated based on new information obtainedfrom the plurality of images, such as newly identified targets, newlyvisible obstacles, newly visible signs, etc.

FIG. 32B illustrates an exemplary braking profile, consistent withdisclosed embodiments. In the example shown in FIG. 32B, the brakingprofile includes four segments. However, the number of segments isexemplary and a braking profile consistent with the disclosedembodiments may include any appropriate number of segments (e.g., 2, 3,4, 5, 6, etc., segments).

In segment 1, system 100 may prepare for a possible braking scenario.For example, in this zone, system 100 may receive information (e.g., mapdata and/or image data) about an upcoming intersection (e.g., anintersection with traffic lights). System 100 may gradually reduce thespeed of vehicle 200 (e.g., the speed of vehicle 200 may be graduallyreduced to about 70-50 km/hr, depending on the characteristics of theintersection). System 100 may cause the speed adjustment, for example,approximately 100-200 meters before the junction at average decelerationaround 0.2-0.4 m/sec² (with a maximum of about 0.5 m/sec²). Bydecreasing the speed of vehicle 200 in segment 1, system 100 may providethe driver and/or passengers of vehicle 200 with confidence that system100 is aware of the approaching intersection.

In segment 2, system 100 may apply strong braking. In this segment,system 100 may receive information based on the analysis of image datathat, for example, a traffic light is red and/or there is a stoppedvehicle at the intersection in the line in which vehicle 200 istraveling. System 100 may cause vehicle 200 to experience significantspeed reduction in this zone, while the distance to the stop line orstopped vehicle is still large. Doing so may provide the driver and/orpassengers of vehicle 200 with a comfortable feeling that system 100will have more than enough space to complete the braking maneuver. Thiszone may be approximately 30-100 meters away from the stop line, and theaverage deceleration may be about 1.5-2.5 m/sec² (with a maximum ofabout 3.5 m/sec²).

In segment 3, system 100 may make a moderate braking adjustment. Forexample, in this segment, system 100 may adjust the speed of vehicle 200according to the remaining distance to the stop line or stopped vehicleand based on the current speed and deceleration of vehicle 200. Bymaking this adjustment, system 100 may provide the driver and/orpassengers of vehicle 200 with an indication that system 100 isreleasing most of the braking power while the speed is still beingreduced but slower than before. This zone may be approximately 5-30meters away from the stop line or stopped vehicle, and the averagedeceleration may be about 0.5-1.5 m/sec² (with a maximum of about 2m/sec²).

In segment 4, system 100 may make a small braking adjustment. In thissegment, system 100 may close the remaining distance to the stop line orstopped vehicle at a very low speed gradually bringing it to zero. Inthis zone, the driver and/or passengers may feel that vehicle 200 isslowly sliding into its position (e.g., at the stop line or behindanother vehicle). This zone may constitute approximately the last 5-7meters to the stop line or stopped vehicle, and the average decelerationmay be about 0.3-0.5 m/sec² (with a maximum of about 1 m/sec²).

FIG. 33 is an exemplary block diagram of memory 140 and/or 150, whichmay store instructions for performing one or more operations consistentwith disclosed embodiments. As illustrated in FIG. 33 , memory 140 maystore one or more modules for performing the trigger detection andresponses described herein. For example, memory 140 may store a triggeridentification module 3300 and an action response module 3310.

Trigger identification module 3300 may store instructions which, whenexecuted by processing unit 110, may detect the presence and/orexistence of a trigger for stopping vehicle 200. For example, triggeridentification module 3300 may process the plurality of images receivedfrom at least one image capture device 122-124 to detect the presence ofa trigger. As discussed above, this may include identifying object 3220,traffic light 3230, and/or traffic sign 3240.

Action response module 3310 may store instructions which, when executedby processing unit 110, may respond to the presence of a trigger. Forexample, action response module 3310 may execute control to decelerateand/or stop vehicle 200. For example, action response module 3310 mayexecute a braking profile to decelerate and/or stop vehicle 200. Theaction response module 3310 may determining to execute one of aplurality of braking profiles based on a number of braking profilesbased on, for example, environmental factors, the type of triggerdetected, and the velocity of vehicle 200. To navigate vehicle 200 to astop based on the detected trigger, processing unit 110 may transmitelectronic signals to one or more of throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200. Further, one ormore actuators may control throttling system 220, braking system 230,and/or steering system 240. For example, processing unit 110 maytransmit electronic signals that cause system 100 to physically depressthe brake by a predetermined amount.

FIG. 34 illustrates a process 3400 for navigating vehicle 200,consistent with disclosed embodiments. Process 3000 may identify atrigger (e.g., an object, vehicle, traffic light, traffic sign, etc.)for stopping vehicle 200 and execute a braking profile in response tothe detection of the trigger. Process 3400 may detect various types oftriggers determine which of a plurality of braking profiles to execute.

The braking profile may be based on the type of a detected triggerand/or other environmental conditions related to the detected trigger(e.g., a distance to an object). For example, in some embodiments, thebraking profile may include a plurality of segments (e.g., threesegments). In a three segment braking profile, a first segment of thebraking profile may be associated with a first deceleration rate, asecond segment of the braking profile may include a second decelerationrate less than the first deceleration rate, and a third segment of thebraking profile may include causing a level of braking to be decreasedas a target stopping location is approached, as determined based on ananalysis of a plurality of images acquired by one or more of imagecapture devices 122-126. For example, in some embodiments, the firstsegment may cause vehicle 200 to progressively increase deceleration,the second segment may cause vehicle 200 to decelerate at a constantrate, and the third segment may cause vehicle 200 to progressivelydecrease its deceleration rate to zero (or near zero).

At step 3410, process 3400 may acquire, using at least one of imagecapture devices 122-126, a plurality of images of an area in thevicinity of vehicle 200. For example, processing unit 110 may receivethe plurality of images may through data interface 128. Processing unit110 may be configured to determine the presence of one or more objectsand/or signs located on or near roadway 3210 based on the acquiredimages. For example, processing unit 110 may be configured to determinethe presence of object 3220, traffic light 3230, and/or traffic sign3240. Processing unit 110 may be further configured to read the contentof traffic sign 3240. For example, processing unit 110 may be configuredto differentiate between a speed limit sign and a stop sign. Further,processing unit 110 may be configured to process images to determine thespeed limit based on traffic sign 3240.

At step 3420, process 300 may include a trigger for stopping vehicle200. For example, processing unit 110 may identify a trigger based onthe presence of object 3220 within the vicinity of vehicle 200.Additionally or alternatively, processing unit 110 may identify atrigger based on a current signal status of traffic light 3230.According to some embodiments, processing unit 110 may determine thattraffic light 3230 constitutes a trigger if it is not green. Accordingto some embodiments, processing unit 110 may determine that traffic sign3240 constitutes a trigger if it is a stop sign, for example. In someembodiments, traffic sign 3240 may constitute a trigger if it is a speedlimit sign and vehicle 200 is exceeding the posted speed limit. Furtherin some embodiments, processing unit 110 may cause system 100 to providean audible announcement identifying the trigger and/or indicating thatvehicle 200 is braking or about to begin braking.

After processing unit 110 identifies the trigger, at step 3430,processing unit 110 may cause vehicle 200 to slow down and/or stopaccording to a braking profile. In some embodiments, the braking profilemay comprise a number of profile segments. The different profilesegments of a braking profile may be defined by a time period and/or avehicle speed. For example, processing unit 110 may execute controls ofvehicle 200 in accordance with a first profile segment for apredetermined period of time or until the speed of vehicle 200 reaches atarget value. The target value may be a predetermined speed or it may bea percentage of the speed that vehicle 200 was traveling prior toexecution of the braking profile. To implement the braking profile,processing unit 110 may transmit electronic signals to one or more ofthrottling system 220, braking system 230, and/or steering system 240 ofvehicle 200. Further, one or more actuators may control throttlingsystem 220, braking system 230, and/or steering system 240.

As discussed above, according to some embodiments, a profile segment ofthe braking profile may correspond to a deceleration rate. For example,a braking profile may include multiple segments, and the first segmentmay be associated with a first deceleration rate. The braking profilemay further include a second segment associated with a seconddeceleration rate. The second deceleration rate may be less than firstdeceleration rate.

Further, as discussed above, according to some embodiments, brakingprofile may include a third profile segment in which a level of brakingis decreased as a function of the distance between vehicle 200 andunderlying factors that give rise to the trigger (e.g., object 3220,traffic light 3230, traffic sign 3240). For example, the braking profilemay include decreasing the level of braking as vehicle 200 approachestarget stopping location 3250. Further, reaching target stoppinglocation 3250 may be accompanied by a release in braking pressure ofvehicle 200.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, or other opticaldrive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A control system for a vehicle, comprising: a camera to acquire a plurality of images of an area ahead of the vehicle; and an image processing system configured to: identify, based on analysis of the plurality of images, a trigger event for stopping the vehicle; and transmit a signal to a braking system to apply a brake to the vehicle, in response to the trigger event, according to a first deceleration rate and a second deceleration rate, the first deceleration rate less than the second deceleration rate, wherein the camera is front-mounted installed in an upper portion of a windshield of the vehicle.
 2. The control system of claim 1, wherein the trigger event includes detecting an object.
 3. The control system of claim 2, wherein the object includes another vehicle in front of the vehicle.
 4. The control system of claim 2, wherein the object includes a pedestrian.
 5. The control system of claim 1, where the in-vehicle processor is configured to provide an audible notification of the trigger.
 6. The control system of claim 1, wherein in response to the signal, the braking system is to brake according to a third deceleration rate.
 7. A control system for a vehicle, comprising: a camera to acquire a plurality of images of an area ahead of the vehicle; and an image processing system configured to: identify, based on analysis of the plurality of images of an area ahead of the vehicle, a trigger event for stopping the vehicle; and transmit a signal to a braking system to apply a first brake to the vehicle, in response to the trigger event, according to a first deceleration rate until a threshold is reached, and apply a second brake to the vehicle according to a second deceleration rate, the first deceleration rate less than the second deceleration rate, wherein the camera is front-mounted installed in an upper portion of a windshield of the vehicle.
 8. The control system of claim 7, wherein the trigger event is a risk of colliding with another vehicle.
 9. The control system of claim 7, wherein in response to the signal, the braking system is to apply a third brake to the vehicle according to a third deceleration rate. 